Why AWS Lambda is at the core of our engine

In today’s fast-paced digital landscape, businesses need to be agile, scalable, and capable of handling ever-increasing volumes of data. AWS Lambda and serverless computing let you run code in response to events without managing servers.

At AppGenie we are working through our own API model utilising locally hosted models, including the ability to dial your own – we call it NeuroSync.

The problem: complex data processing at scale

As businesses grow, so do the complexities of their data. In our case, we were faced with the challenge of managing high volumes of requests, processing user data, and interacting with multiple external systems in a seamless, efficient manner. All of this while ensuring reliability and keeping things scalable and cost-effective.

Our solution is built around AWS Lambda. Its serverless nature means we don’t have to worry about provisioning servers or managing infrastructure. Instead, we focus on what matters – writing the business logic, handling asynchronous tasks, and enabling real-time interactions.

The solution: a streamlined workflow for automated, scalable processing

To show how we leverage Lambda and other AWS services, here’s the core of the system we built. It is designed to efficiently handle complex data processing without sacrificing performance.

Step 1: The SQS message event

We begin by receiving a message from an SQS (Simple Queue Service) event. The message contains user input, system information, and metadata, which are all critical to processing the task. SQS acts as a buffer between data producers and consumers, allowing us to process messages asynchronously and avoid overload.

public static string StoreRequest(Request request)
{
    LLMRequest lLMRequest = new LLMRequest(request);
    using var sqsClient = new AmazonSQSClient();

    var sendMessageRequest = new SendMessageRequest
    {
        QueueUrl = Credentials.SQS_QUEUE_REQUEST,
        MessageBody = JsonSerializer.Serialize(lLMRequest)
    };

    Task result = sqsClient.SendMessageAsync(sendMessageRequest);
    result.Wait();

    return lLMRequest.RequestId;
}
                        

Here, we send a request to an SQS queue, which acts as our message broker, decoupling the system and allowing for better scalability. The result is that we can handle many requests simultaneously, without having to worry about bottlenecks.

Step 2: Intent classification and Salesforce integration

Once the message arrives at the Lambda function, we begin processing it by extracting the necessary data. This involves classifying the user’s intent, which is a vital step for determining how we should handle the request. In this case, we use AWS Comprehend, a fully managed natural language processing service that can easily analyse and classify text.

Once the intent is identified, we interact with Salesforce to fetch the relevant data. Salesforce handles large volumes of CRM data and exposes it via APIs, which we call in real time.

var ragProvider = RAG.ProviderFactory.Create("Salesforce");
JsonElement sfdcData = await ragProvider.GetRagDataAsync(request, sfdc);
                        

This step is crucial because it allows us to dynamically retrieve the necessary Salesforce data in real time, ensuring we always have the most up-to-date information feeding into NeuroSync.

Step 3: Storing data on S3 for long-term access

Once the data is retrieved and processed, we need to store it somewhere that’s both durable and easily accessible. This is where Amazon S3 comes into play. S3 is a scalable storage solution that is perfect for storing all kinds of data, from JSON blobs to user-generated content.

string s3Path = $"/{request.Client.Name}/{request.Client.User}/{request.Client.Instance}/{request.ConversationId}/{request.RequestId}";
await ArtifactUploader.UploadAsync(s3Path, prompt, intents, sfdcData);
                        

Using ArtifactUploader, we upload the processed data and any generated prompts or intents into S3. With S3’s object versioning and lifecycle policies, we ensure the data is safe, long-lasting, and available when we need to revisit requests or audit behaviour.

Step 4: Triggering a follow-up request

Finally, we initiate follow-up processes based on the results of the initial processing. This is done via another SQS event, keeping the system modular and scalable.

AWS.SQS.StoreRequest(AWS.Credentials.SQS_QUEUE_RAG, request);
                        

This approach enables us to handle complex workflows without blocking or delaying critical tasks. Lambda scales up as needed and scales back down to zero when idle.

Why AWS Lambda and serverless is the future

There are clear reasons we keep coming back to Lambda and serverless for this kind of workload:

• Scalability: Lambda automatically scales with demand. We focus on functionality, not capacity planning.
• Cost-effectiveness: Pay-as-you-go pricing means we only pay for actual compute time, not idle servers.
• Event-driven architecture: Lambda integrates with SQS, SNS, S3, and more, making it ideal for event-driven designs.
• High availability: Lambda runs across multiple availability zones in a region, giving us resilience by default.
• Seamless integration: Lambda connects cleanly with Salesforce, Comprehend, S3, and our own NeuroSync stack.

Making it work for your business

By combining the power of AWS Lambda, Salesforce, and S3, we have built a highly efficient, scalable, and cost-effective solution for processing complex data at scale. Whether you are handling customer records, processing user input, or running deep AI models, these technologies can streamline your workflows and give you a competitive edge.

With our tailored solutions and expertise in serverless technologies, we can help your business stay ahead of the curve. We can show you how to leverage these tools to drive business growth, improve operational efficiency, and deliver better user experiences.

Final thoughts

Serverless technologies like AWS Lambda, SQS, and S3 are the future of scalable data processing. With NeuroSync, we leverage these tools to build systems that can process complex data at speed, without the overhead of traditional infrastructure management.

Whether you need seamless integrations with Salesforce, real-time processing of user data, or cost-effective storage for huge volumes of information, we have the expertise to make it happen.

Ready to take your business to the next level with serverless computing? Let’s chat.

• Back to Blog