{"id":166,"date":"2016-03-31T13:03:02","date_gmt":"2016-03-31T07:33:02","guid":{"rendered":"https:\/\/www.nocturnalknight.com\/?p=166"},"modified":"2016-03-31T13:03:02","modified_gmt":"2016-03-31T07:33:02","slug":"developing-a-fast-and-big-data-acquisition-system-with-a-near-real-time-analytics-part-1","status":"publish","type":"post","link":"http:\/\/3.10.118.248\/?p=166","title":{"rendered":"Developing a Fast and Big data Acquisition System with a near Real Time Analytics &#8211; Part 1"},"content":{"rendered":"<p><strong>Introduction<\/strong><br \/>\nThis series of How To or Case Study I am attempting to write was the result of the work of our team for the past 2+ years.We were developing a <strong>Predictive Analytics Platform<\/strong> for a global truck OEM. This was to be integrated with their live <strong>OBU data, Warranty, Research &amp; Design, Customer Support, CRM and DMS<\/strong> among other things.<br \/>\nIn this journey, we have attempted to solve the problem in incremental steps. Currently we are working on the predictive analytics with learning workflows. So, I believe Its time to pen down the experience with building the other 3 incremental solutions.<\/p>\n<ol>\n<li><strong>\u00a0<\/strong><em><strong>First Baby Step<\/strong> <\/em>&#8211; Fast Data capture and Conventional Analytics &#8211;\n<ul>\n<li>Kafka, Redis, PostreSQL<\/li>\n<\/ul>\n<\/li>\n<li><em><strong>Next Logical Step<\/strong> &#8211; <\/em>Big Data capture, Warehousing and Conventional Analytics\n<ul>\n<li>Kafka, Storm\/Spark, Hadoop\/Hive, Zookeeper<\/li>\n<\/ul>\n<\/li>\n<li><em><strong>The Bulls Eye<\/strong> &#8211; <\/em>Real Time Analytics on Big-Data\n<ul>\n<li>Same as Above with Solr and Zeppelin<\/li>\n<\/ul>\n<\/li>\n<li><strong><em>The Holy Grail<\/em><\/strong> &#8211; Predictive Analytics\n<ul>\n<li>Same as Above with MLib on Spark<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<p>Now, in this post I will write about <em>&#8220;The First Baby Step&#8221;. T<\/em>his\u00a0 involves fast acquisition of data, Real-time analytics and long term data archival.<br \/>\nThe disparate data sets and sources posed a significant complexity, not to mention the myriad polling frequencies, sync models and EOD jobs.\u00a0 It goes without saying that the #OEM had a significant investment in SAP infrastructure. We had studied multiple architecture models, (Some are available in this <a href=\"http:\/\/35.177.175.1.xip.io\/wp-content\/uploads\/2016\/03\/Reference.Architecture.SAP_Hortonworks.v1.4-June-2014.pdf\">Reference Architecture Model from Horton Works and SAP<\/a>)<br \/>\nThe following are the considerations from the data perspective,<\/p>\n<ol>\n<li>FastData &#8211; Realtime Telematics data from the OBU.<\/li>\n<li>BigData &#8211; Diagonastics data from each truck had 40+ parameters and initial pilot of 7500 trucks.<\/li>\n<li>Structured Data &#8211; Data from Dealer Management System and Customer Relationship Management System.<\/li>\n<li>Transactional Data &#8211; Data from Warranty management and Customer Support systems.<\/li>\n<\/ol>\n<p><strong>Fast Data:<\/strong> Our primary challenge for the 1st phase of design\/development was the scaling of the data acquisition system to collect data from thousands of nodes, each of which sent 40 sensor readings polled once per second and transmitted every 6 seconds once. While maintaining the ability to query the data in real time for event detection. While each data record was only ~300kb, our expected maximum sensor load indicated a collection rate of about 27 million records, or 22.5GB, per hour. However, our primary issue was not data size, but data rate. A large number of inserts had to happen each second, and we were unable to buffer inserts into batches or transactions without incurring a delay in the real-time data stream.<br \/>\nWhen designing network applications, one must consider the two canonical I\/O bottlenecks: <strong>Network I\/O<\/strong>, and <strong>Filesystem I\/O<\/strong>. For our use case, we had little influence over network I\/O speeds. We had no control over the locations where our truck sensors would be at any given time, or in the bandwidth or network infrastructure of said location (Our OBDs communicated using GPRS on GSM Network). With network latency as a known variant, we focused on addressing the bottleneck we could control: Filesystem I\/O. For the immediate collection problem, this means we evaluated databases to insert the data into as it was collected. While we initially attempted to collect the data in a relational database (PostgreSQL), we soon discovered that while PostgreSQL could potentially handle the number of inserts per second, it was unable to respond to read queries simultaneously. Simply put, we were unable to read data while we were collecting it, preventing us from doing any real-time analysis (or any analysis at all, for that matter, unless we stopped data collection).<br \/>\nThe easiest way to avoid slowdowns due to disk operations is to avoid the disk altogether, we mitigated this by leveraging <a href=\"http:\/\/www.redis.io\">Redis<\/a>, an open-source in-memory NoSQL datastore. Redis stores all data in RAM and in hybrid models in Flash storage (like an SSD) allowing lightning fast reads and writes. With Redis, we were easily able to insert all of our collected data as it was transmitted from the sensor nodes, and query the data simultaneously for event detection and analytics. In fact, were were also able to leverage Pub\/Sub functionality on the same Redis server to publish notifications of detected events for transmission to event driven workers, without any performance issues.<br \/>\nIn addition to speed, Redis features advanced data structures, including Lists, Sets, Hashes,Geospatials and Sorted Sets, rather than the somewhat limiting key\/value pair consistent with many NoSQL stores.<br \/>\n<figure id=\"attachment_178\" aria-describedby=\"caption-attachment-178\" style=\"width: 300px\" class=\"wp-caption alignright\"><a href=\"http:\/\/35.177.175.1.xip.io\/wp-content\/uploads\/2016\/03\/redis-data-structures.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-178 size-medium\" src=\"http:\/\/35.177.175.1.xip.io\/wp-content\/uploads\/2016\/03\/redis-data-structures-300x149.png\" alt=\"Available Data Structures in Redis\" width=\"300\" height=\"149\" srcset=\"http:\/\/3.10.118.248\/wp-content\/uploads\/2016\/03\/redis-data-structures-300x149.png 300w, http:\/\/3.10.118.248\/wp-content\/uploads\/2016\/03\/redis-data-structures.png 743w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><figcaption id=\"caption-attachment-178\" class=\"wp-caption-text\">Available Data Structures in Redis<\/figcaption><\/figure><br \/>\n<strong>Sorted Sets<\/strong> proved to be an excellent data structure to model timeseries data, by setting the score to the timestamp of a given datapoint. This automatically ordered our timeseries\u2019, even when data was inserted out of order, and allowed querying by timestamp, timestamp range, or by \u201cmost recent #\u201d of records (which is merely the last # values of the set).<br \/>\nOur use case requires us to archive our data for a period of time, enabling the business users to run a historical analytics along with data from the real-time source.<br \/>\nEnter <em><strong>Data Temperatures, <\/strong><\/em><br \/>\n<figure id=\"attachment_180\" aria-describedby=\"caption-attachment-180\" style=\"width: 450px\" class=\"wp-caption alignright\"><a href=\"http:\/\/35.177.175.1.xip.io\/wp-content\/uploads\/2016\/03\/Data-Temperatures.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-180\" src=\"http:\/\/35.177.175.1.xip.io\/wp-content\/uploads\/2016\/03\/Data-Temperatures-300x126.jpg\" alt=\"Data temperatures\" width=\"450\" height=\"189\" srcset=\"http:\/\/3.10.118.248\/wp-content\/uploads\/2016\/03\/Data-Temperatures-300x126.jpg 300w, http:\/\/3.10.118.248\/wp-content\/uploads\/2016\/03\/Data-Temperatures.jpg 637w\" sizes=\"auto, (max-width: 450px) 100vw, 450px\" \/><\/a><figcaption id=\"caption-attachment-180\" class=\"wp-caption-text\">Data Temperatures<\/figcaption><\/figure><br \/>\n<strong>Hot Data <\/strong>&#8211; The data which is frequently accessed and is currently being polled\/gathered.<br \/>\n<strong>Warm Data<\/strong> &#8211; The data which is currently not being polled but still frequently used.<br \/>\n<strong>Cold Data<\/strong> &#8211; The data that is in warehouse-mode, but still can be accessed for BI or analytics jobs with a bit of I\/O Overhead.<br \/>\nSince Redis keeps all data in RAM that is the HOT Area, our Redis datastore was only able to hold as much data as the server had &#8220;Available RAM&#8221;. Our data, inserted at a rate of 27GB\/hour, quickly outgrew this limitation. To scale this solution and archive our data for future analysis, we set up an automated migration script to push the oldest data in our Redis datastore to a PostgreSQL database with more storage scalability. As explained above, since Redis has native data types for Time Series data, it was a simple enough process for the Load operation.<br \/>\nThe other consideration to be exercised is the &#8220;Available RAM&#8221;. As the amount of data that is queried, CPU cycles used and the RAM used for the Processing determines the amount of memory available for data stores. be reminded if the data-stores are fill to the brim your processing job is going to utulise the disk I\/O. Which is very bad.<br \/>\nWe wrote a REST API as an interface to our two datastores allowing client applications a unified query interface, without having to worry about which data-store a particular piece of data resided in. This web-service layer defined the standards for the time, range and parameters.<br \/>\n<figure id=\"attachment_181\" aria-describedby=\"caption-attachment-181\" style=\"width: 730px\" class=\"wp-caption alignright\"><a href=\"http:\/\/35.177.175.1.xip.io\/wp-content\/uploads\/2016\/03\/Fast-Data-Arch.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-large wp-image-181\" src=\"http:\/\/35.177.175.1.xip.io\/wp-content\/uploads\/2016\/03\/Fast-Data-Arch-1024x576.png\" alt=\"Fast Data Architecture with Redis and Kafka\" width=\"730\" height=\"411\" srcset=\"http:\/\/3.10.118.248\/wp-content\/uploads\/2016\/03\/Fast-Data-Arch-1024x576.png 1024w, http:\/\/3.10.118.248\/wp-content\/uploads\/2016\/03\/Fast-Data-Arch-300x169.png 300w, http:\/\/3.10.118.248\/wp-content\/uploads\/2016\/03\/Fast-Data-Arch-768x432.png 768w, http:\/\/3.10.118.248\/wp-content\/uploads\/2016\/03\/Fast-Data-Arch.png 1280w\" sizes=\"auto, (max-width: 730px) 100vw, 730px\" \/><\/a><figcaption id=\"caption-attachment-181\" class=\"wp-caption-text\">Fast Data Architecture with Redis and Kafka\u00a0\u00a0<\/figcaption><\/figure><br \/>\nWith the above represented architecture in place, generating automated event detection and real-time notifications was feasible, again through the use of Redis. Since Redis also offers Pub\/Sub functionality, we were able to monitor incoming data in Redis using a small service, and push noteworthy events to a notification channel on the same Redis server, from which subscribed SMTP workers could send out notifications in real-time. This can even be channeled to an MQ\/ESB or any Asynchronous mechanism to initiate actions or reactions.<br \/>\n&nbsp;<br \/>\nOur experiences show Kafka and Redis to be a powerful tool for Big Data applications, specifically for high-throughput data collection. The benefits of Kafka as a collection mechanism, coupled with inmemory data storage using Redis and data migration to a deep analytics platform, such as relational databases or even Hadoop\u2019s HDFS, yields a powerful and versatile architecture suitable for many Big Data applications.<br \/>\nAfter we have implemented HDFS and Spark in Phase 2-3 of this roadmap, we have of-course configured redis in the said role. Hope I have covered enough of the 1st step in our Big-Data journey. Will write an article per week regarding the other 3 phases we have implemented successfully.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction This series of How To or Case Study I am attempting to write was the result of the work of our team for the past 2+ years.We were developing a Predictive Analytics Platform for a global truck OEM. This was to be integrated with their live OBU data, Warranty, Research &amp; Design, Customer Support, &hellip; <\/p>\n<p class=\"read-more\"><a class=\"btn btn-default\" href=\"http:\/\/3.10.118.248\/?p=166\"> Read More<span class=\"screen-reader-text\">  Read More<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"image","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"footnotes":""},"categories":[1],"tags":[408],"class_list":["post-166","post","type-post","status-publish","format-image","hentry","category-uncategorized","tag-oem","post_format-post-format-image"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.0.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"Introduction This series of How To or Case Study I am attempting to write was the result of the work of our team for the past 2+ years.We were developing a Predictive Analytics Platform for a global truck OEM. This was to be integrated with their live OBU data, Warranty, Research &amp; Design, Customer Support,\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Ramkumar Sundarakalatharan\"\/>\n\t<link rel=\"canonical\" href=\"http:\/\/3.10.118.248\/?p=166\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.0.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_GB\" \/>\n\t\t<meta property=\"og:site_name\" content=\"Nocturnalknight&#039;s Lair - Observations of a Random Wanderer!\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"Developing a Fast and Big data Acquisition System with a near Real Time Analytics \u2013 Part 1 - Nocturnalknight&#039;s Lair\" \/>\n\t\t<meta property=\"og:description\" content=\"Introduction This series of How To or Case Study I am attempting to write was the result of the work of our team for the past 2+ years.We were developing a Predictive Analytics Platform for a global truck OEM. This was to be integrated with their live OBU data, Warranty, Research &amp; Design, Customer Support,\" \/>\n\t\t<meta property=\"og:url\" content=\"http:\/\/3.10.118.248\/?p=166\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2016-03-31T07:33:02+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2016-03-31T07:33:02+00:00\" \/>\n\t\t<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n\t\t<meta name=\"twitter:site\" content=\"@nocturnalknight\" \/>\n\t\t<meta name=\"twitter:title\" content=\"Developing a Fast and Big data Acquisition System with a near Real Time Analytics \u2013 Part 1 - Nocturnalknight&#039;s Lair\" \/>\n\t\t<meta name=\"twitter:description\" content=\"Introduction This series of How To or Case Study I am attempting to write was the result of the work of our team for the past 2+ years.We were developing a Predictive Analytics Platform for a global truck OEM. This was to be integrated with their live OBU data, Warranty, Research &amp; Design, Customer Support,\" \/>\n\t\t<meta name=\"twitter:creator\" content=\"@nocturnalknight\" \/>\n\t\t<script type=\"application\/ld+json\" class=\"aioseo-schema\">\n\t\t\t{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"BlogPosting\",\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?p=166#blogposting\",\"name\":\"Developing a Fast and Big data Acquisition System with a near Real Time Analytics \\u2013 Part 1 - Nocturnalknight's Lair\",\"headline\":\"Developing a Fast and Big data Acquisition System with a near Real Time Analytics &#8211; Part 1\",\"author\":{\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?author=2#author\"},\"publisher\":{\"@id\":\"http:\\\/\\\/3.10.118.248\\\/#organization\"},\"image\":{\"@type\":\"ImageObject\",\"url\":\"http:\\\/\\\/35.177.175.1.xip.io\\\/wp-content\\\/uploads\\\/2016\\\/03\\\/redis-data-structures-300x149.png\",\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?p=166\\\/#articleImage\"},\"datePublished\":\"2016-03-31T13:03:02+01:00\",\"dateModified\":\"2016-03-31T13:03:02+01:00\",\"inLanguage\":\"en-GB\",\"mainEntityOfPage\":{\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?p=166#webpage\"},\"isPartOf\":{\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?p=166#webpage\"},\"articleSection\":\"Uncategorized, OEM, Image\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?p=166#breadcrumblist\",\"itemListElement\":[{\"@type\":\"ListItem\",\"@id\":\"http:\\\/\\\/3.10.118.248#listItem\",\"position\":1,\"name\":\"Home\",\"item\":\"http:\\\/\\\/3.10.118.248\",\"nextItem\":{\"@type\":\"ListItem\",\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?cat=1#listItem\",\"name\":\"Uncategorized\"}},{\"@type\":\"ListItem\",\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?cat=1#listItem\",\"position\":2,\"name\":\"Uncategorized\",\"item\":\"http:\\\/\\\/3.10.118.248\\\/?cat=1\",\"nextItem\":{\"@type\":\"ListItem\",\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?p=166#listItem\",\"name\":\"Developing a Fast and Big data Acquisition System with a near Real Time Analytics &#8211; Part 1\"},\"previousItem\":{\"@type\":\"ListItem\",\"@id\":\"http:\\\/\\\/3.10.118.248#listItem\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?p=166#listItem\",\"position\":3,\"name\":\"Developing a Fast and Big data Acquisition System with a near Real Time Analytics &#8211; Part 1\",\"previousItem\":{\"@type\":\"ListItem\",\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?cat=1#listItem\",\"name\":\"Uncategorized\"}}]},{\"@type\":\"Organization\",\"@id\":\"http:\\\/\\\/3.10.118.248\\\/#organization\",\"name\":\"Nocturnalknight's Lair\",\"description\":\"Observations of a Random Wanderer!\",\"url\":\"http:\\\/\\\/3.10.118.248\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"http:\\\/\\\/3.10.118.248\\\/wp-content\\\/uploads\\\/2026\\\/08\\\/cropped-Ram-Profile.avif\",\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?p=166\\\/#organizationLogo\"},\"image\":{\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?p=166\\\/#organizationLogo\"},\"sameAs\":[\"https:\\\/\\\/x.com\\\/nocturnalknight\",\"https:\\\/\\\/www.linkedin.com\\\/in\\\/nocturnalknight\\\/\"]},{\"@type\":\"Person\",\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?author=2#author\",\"url\":\"http:\\\/\\\/3.10.118.248\\\/?author=2\",\"name\":\"Ramkumar Sundarakalatharan\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?p=166#authorImage\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/818bc4a4d5681de6957f83aca2601d598459bf37a0a8b17d5abb1a889e2b9298?s=96&d=mm&r=g\",\"width\":96,\"height\":96,\"caption\":\"Ramkumar Sundarakalatharan\"}},{\"@type\":\"WebPage\",\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?p=166#webpage\",\"url\":\"http:\\\/\\\/3.10.118.248\\\/?p=166\",\"name\":\"Developing a Fast and Big data Acquisition System with a near Real Time Analytics \\u2013 Part 1 - Nocturnalknight's Lair\",\"description\":\"Introduction This series of How To or Case Study I am attempting to write was the result of the work of our team for the past 2+ years.We were developing a Predictive Analytics Platform for a global truck OEM. This was to be integrated with their live OBU data, Warranty, Research & Design, Customer Support,\",\"inLanguage\":\"en-GB\",\"isPartOf\":{\"@id\":\"http:\\\/\\\/3.10.118.248\\\/#website\"},\"breadcrumb\":{\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?p=166#breadcrumblist\"},\"author\":{\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?author=2#author\"},\"creator\":{\"@id\":\"http:\\\/\\\/3.10.118.248\\\/?author=2#author\"},\"datePublished\":\"2016-03-31T13:03:02+01:00\",\"dateModified\":\"2016-03-31T13:03:02+01:00\"},{\"@type\":\"WebSite\",\"@id\":\"http:\\\/\\\/3.10.118.248\\\/#website\",\"url\":\"http:\\\/\\\/3.10.118.248\\\/\",\"name\":\"Nocturnalknight's Lair\",\"description\":\"Observations of a Random Wanderer!\",\"inLanguage\":\"en-GB\",\"publisher\":{\"@id\":\"http:\\\/\\\/3.10.118.248\\\/#organization\"}}]}\n\t\t<\/script>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"Developing a Fast and Big data Acquisition System with a near Real Time Analytics \u2013 Part 1 - Nocturnalknight's Lair","description":"Introduction This series of How To or Case Study I am attempting to write was the result of the work of our team for the past 2+ years.We were developing a Predictive Analytics Platform for a global truck OEM. This was to be integrated with their live OBU data, Warranty, Research & Design, Customer Support,","canonical_url":"http:\/\/3.10.118.248\/?p=166","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"BlogPosting","@id":"http:\/\/3.10.118.248\/?p=166#blogposting","name":"Developing a Fast and Big data Acquisition System with a near Real Time Analytics \u2013 Part 1 - Nocturnalknight's Lair","headline":"Developing a Fast and Big data Acquisition System with a near Real Time Analytics &#8211; Part 1","author":{"@id":"http:\/\/3.10.118.248\/?author=2#author"},"publisher":{"@id":"http:\/\/3.10.118.248\/#organization"},"image":{"@type":"ImageObject","url":"http:\/\/35.177.175.1.xip.io\/wp-content\/uploads\/2016\/03\/redis-data-structures-300x149.png","@id":"http:\/\/3.10.118.248\/?p=166\/#articleImage"},"datePublished":"2016-03-31T13:03:02+01:00","dateModified":"2016-03-31T13:03:02+01:00","inLanguage":"en-GB","mainEntityOfPage":{"@id":"http:\/\/3.10.118.248\/?p=166#webpage"},"isPartOf":{"@id":"http:\/\/3.10.118.248\/?p=166#webpage"},"articleSection":"Uncategorized, OEM, Image"},{"@type":"BreadcrumbList","@id":"http:\/\/3.10.118.248\/?p=166#breadcrumblist","itemListElement":[{"@type":"ListItem","@id":"http:\/\/3.10.118.248#listItem","position":1,"name":"Home","item":"http:\/\/3.10.118.248","nextItem":{"@type":"ListItem","@id":"http:\/\/3.10.118.248\/?cat=1#listItem","name":"Uncategorized"}},{"@type":"ListItem","@id":"http:\/\/3.10.118.248\/?cat=1#listItem","position":2,"name":"Uncategorized","item":"http:\/\/3.10.118.248\/?cat=1","nextItem":{"@type":"ListItem","@id":"http:\/\/3.10.118.248\/?p=166#listItem","name":"Developing a Fast and Big data Acquisition System with a near Real Time Analytics &#8211; Part 1"},"previousItem":{"@type":"ListItem","@id":"http:\/\/3.10.118.248#listItem","name":"Home"}},{"@type":"ListItem","@id":"http:\/\/3.10.118.248\/?p=166#listItem","position":3,"name":"Developing a Fast and Big data Acquisition System with a near Real Time Analytics &#8211; Part 1","previousItem":{"@type":"ListItem","@id":"http:\/\/3.10.118.248\/?cat=1#listItem","name":"Uncategorized"}}]},{"@type":"Organization","@id":"http:\/\/3.10.118.248\/#organization","name":"Nocturnalknight's Lair","description":"Observations of a Random Wanderer!","url":"http:\/\/3.10.118.248\/","logo":{"@type":"ImageObject","url":"http:\/\/3.10.118.248\/wp-content\/uploads\/2026\/08\/cropped-Ram-Profile.avif","@id":"http:\/\/3.10.118.248\/?p=166\/#organizationLogo"},"image":{"@id":"http:\/\/3.10.118.248\/?p=166\/#organizationLogo"},"sameAs":["https:\/\/x.com\/nocturnalknight","https:\/\/www.linkedin.com\/in\/nocturnalknight\/"]},{"@type":"Person","@id":"http:\/\/3.10.118.248\/?author=2#author","url":"http:\/\/3.10.118.248\/?author=2","name":"Ramkumar Sundarakalatharan","image":{"@type":"ImageObject","@id":"http:\/\/3.10.118.248\/?p=166#authorImage","url":"https:\/\/secure.gravatar.com\/avatar\/818bc4a4d5681de6957f83aca2601d598459bf37a0a8b17d5abb1a889e2b9298?s=96&d=mm&r=g","width":96,"height":96,"caption":"Ramkumar Sundarakalatharan"}},{"@type":"WebPage","@id":"http:\/\/3.10.118.248\/?p=166#webpage","url":"http:\/\/3.10.118.248\/?p=166","name":"Developing a Fast and Big data Acquisition System with a near Real Time Analytics \u2013 Part 1 - Nocturnalknight's Lair","description":"Introduction This series of How To or Case Study I am attempting to write was the result of the work of our team for the past 2+ years.We were developing a Predictive Analytics Platform for a global truck OEM. This was to be integrated with their live OBU data, Warranty, Research & Design, Customer Support,","inLanguage":"en-GB","isPartOf":{"@id":"http:\/\/3.10.118.248\/#website"},"breadcrumb":{"@id":"http:\/\/3.10.118.248\/?p=166#breadcrumblist"},"author":{"@id":"http:\/\/3.10.118.248\/?author=2#author"},"creator":{"@id":"http:\/\/3.10.118.248\/?author=2#author"},"datePublished":"2016-03-31T13:03:02+01:00","dateModified":"2016-03-31T13:03:02+01:00"},{"@type":"WebSite","@id":"http:\/\/3.10.118.248\/#website","url":"http:\/\/3.10.118.248\/","name":"Nocturnalknight's Lair","description":"Observations of a Random Wanderer!","inLanguage":"en-GB","publisher":{"@id":"http:\/\/3.10.118.248\/#organization"}}]},"og:locale":"en_GB","og:site_name":"Nocturnalknight's Lair - Observations of a Random Wanderer!","og:type":"article","og:title":"Developing a Fast and Big data Acquisition System with a near Real Time Analytics \u2013 Part 1 - Nocturnalknight's Lair","og:description":"Introduction This series of How To or Case Study I am attempting to write was the result of the work of our team for the past 2+ years.We were developing a Predictive Analytics Platform for a global truck OEM. This was to be integrated with their live OBU data, Warranty, Research &amp; Design, Customer Support,","og:url":"http:\/\/3.10.118.248\/?p=166","article:published_time":"2016-03-31T07:33:02+00:00","article:modified_time":"2016-03-31T07:33:02+00:00","twitter:card":"summary_large_image","twitter:site":"@nocturnalknight","twitter:title":"Developing a Fast and Big data Acquisition System with a near Real Time Analytics \u2013 Part 1 - Nocturnalknight's Lair","twitter:description":"Introduction This series of How To or Case Study I am attempting to write was the result of the work of our team for the past 2+ years.We were developing a Predictive Analytics Platform for a global truck OEM. This was to be integrated with their live OBU data, Warranty, Research &amp; Design, Customer Support,","twitter:creator":"@nocturnalknight"},"aioseo_meta_data":{"post_id":"166","title":null,"description":null,"keywords":null,"keyphrases":null,"focus_keyword":null,"additional_keywords":null,"truseo_locale":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_custom_url":null,"og_image_custom_fields":null,"og_image_url":null,"og_image_width":null,"og_image_height":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_image_url":null,"twitter_title":null,"twitter_description":null,"schema_type":"default","schema_type_options":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"limit_modified_date":false,"ai":null,"breadcrumb_settings":null,"seo_analyzer_scan_date":null,"created":"2026-08-19 11:25:38","updated":"2026-08-19 11:25:38"},"aioseo_breadcrumb":"<div class=\"aioseo-breadcrumbs\"><span class=\"aioseo-breadcrumb\">\n\t\t\t<a href=\"http:\/\/3.10.118.248\" title=\"Home\">Home<\/a>\n\t\t<\/span><span class=\"aioseo-breadcrumb-separator\">&raquo;<\/span><span class=\"aioseo-breadcrumb\">\n\t\t\t<a href=\"http:\/\/3.10.118.248\/?cat=1\" title=\"Uncategorized\">Uncategorized<\/a>\n\t\t<\/span><span class=\"aioseo-breadcrumb-separator\">&raquo;<\/span><span class=\"aioseo-breadcrumb\">\n\t\t\tDeveloping a Fast and Big data Acquisition System with a near Real Time Analytics \u2013 Part 1\n\t\t<\/span><\/div>","aioseo_breadcrumb_json":[{"label":"Home","link":"http:\/\/3.10.118.248"},{"label":"Uncategorized","link":"http:\/\/3.10.118.248\/?cat=1"},{"label":"Developing a Fast and Big data Acquisition System with a near Real Time Analytics &#8211; Part 1","link":"http:\/\/3.10.118.248\/?p=166"}],"amp_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"http:\/\/3.10.118.248\/index.php?rest_route=\/wp\/v2\/posts\/166","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/3.10.118.248\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/3.10.118.248\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/3.10.118.248\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/3.10.118.248\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=166"}],"version-history":[{"count":0,"href":"http:\/\/3.10.118.248\/index.php?rest_route=\/wp\/v2\/posts\/166\/revisions"}],"wp:attachment":[{"href":"http:\/\/3.10.118.248\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=166"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/3.10.118.248\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=166"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/3.10.118.248\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=166"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}