Large Data Propagation

Enterprises use varied set of systems and each system has its own data. As life goes digital and with social media boom there is data explosion. Also, data is in different states – static in some cases and streaming in others.

Companies are facing challenges with this inconsistent data. Large data needs to be moved in uni / bidirectional ways in a reliable, fast and secure mode across systems so that customer facing teams can respond in time to their queries and concerns; and a quick reliable voice of the enterprise always makes customers happy.

Current Situation

The existing Data Synchronization tools have latency issues and typically the data synchronization is an end-of-the-day process. With this, there is always a lag of one day and in certain cases where the volumes are higher, the process can even take more than a day to complete.

Applications connect to heterogeneous back-ends, and data is stored in legacy systems – Hierarchical databases, RDBMS, files, mainframe systems, etc. Data structures that need to be synchronized require complex mappings between sources and targets, as well as aggregations, calculations, etc.

The solution

Large Data Propagation Tool addresses all the current day business problems, using the proven Storm technology for reliable and faster synchronization through grid computing. The tool is simple to use and can be configured to synchronize data offline / online. The administrator has to simply modify the configuration file and do a setup and scheduling can be done to run at any point or all through. Some salient features of the solution include:
• Fault-tolerance: Storm cluster takes care of workers going down, reassigning tasks as necessary
• Scalability: Add more machines to the cluster and Storm will reassign tasks as per new machines’ availability
• Reliability: All messages are guaranteed to be processed at least once. In case of errors, they might get processed more than once, but will never be left out unattended
• Speed: It’s one of the default factors driving Storm’s design
• Transactional: One messaging semantics for pretty much any computation

For more information on our Large Data Propagation services