IoT devices generate a lot of data. For instance, say you’re dealing with an industrial device sending data like temperature levels continuously to the server, and you’re required to keep all this data stored for a long time.

So, it’s very important to consider how you capture & store this data and make sure that the data is stored in such a way that it can be used for analysis & various BI reporting—then machine learning models can also be implemented on top of this data.

This seems easy when you have a single device to manage, but how should one go about managing a continuous data stream from thousands of IoT devices?

Here’s how-

• Provision enough storage: Looking at the magnitude & volume of data, it’s imperative that you provision enough storage considering its performance parameters while capturing and storing the data. This is especially important because over time this data is going to huge by volume, and certain parameters need to be precomputed as you get the data, store it, and use it for analysis.
• Think Serverless: A serverless architecture is preferred for such scenarios so that whenever there are spikes, your cloud infrastructure can handle it seamlessly.
• Consider an Event-driven architecture: In an even-driven architecture whenever there is an event in the field, your device sends the data to the server, and your server processes it. i.e., server activity spikes are driven by a specific event taking place in the field.

Typically, serverless technologies like Azure functions & AWS Lambda will help you design your architecture to handle voluminous data and activity spikes in an optimum way.

Given below are some recommendations to equip you with the right technology stack for building software for IoT:

• Firmware: From an IoT standpoint, the firmware on your device, typically the OSS can be written using Python or Cross compilers like ESP8266 platforms.
• Server side: On the server side, you will receive events from the firmware into some sort of queue (like MQTT). Technologies like C#, Node.js, Python or similar server-side event driven technologies which can handle your data with ease should be the ideal choice.
• Data: Data should be typically stored in NoSQL DBs to account for a potential change in data structures. Databases like MongoDB, Firebase real-time database, or Cosmos DB should be utilized to store the initial data (i.e., the initial data captured from the devices). Apart from that, the precomputed data that will be used for analysis should be stored in relational databases like Matrix.
• For interfaces- if it’s an enterprise UI, it could have a web interface designed in Angular or similar technologies. If it’s a mobile app, where users must interact with the devices, then Flutter or native development should be considered.
• Incorporating other technologies: If you’re designing skills like Amazon Alexa skills or Google home skills, those should be developed into the respective API platforms & your server-side technology will help them communicate with the devices.
• Analytical tools – Once IoT solution starts getting adopted, users will keep using it in diverse ways. Knowing how users use IoT systems and improve user experience with generated data is a key to adoption. It will be especially important to analyze and draw meaningful inferences from the generated data. BI tools and Machine learning models can be used to get most out of this data.