Discover why innovation is a key driver of technology thought leadership in an exclusive interview covered with Arnab Sen, Tredence.
Arnab, kindly brief us about yourself and your journey as the VP of Data Engineering at Tredence Inc.
As the VP of Data Engineering at Tredence Inc., my journey has been a rewarding evolution. Starting my tenure six to seven years ago as a senior manager, I recognized early on that technology, particularly data engineering, could significantly distinguish us from our competitors. Brought on board to boost our technology prowess, I helped establish our initial technology practice. My relocation to the United States as Director of Data Engineering marked our focused effort to create a distinct niche in data engineering.
Since then, we’ve expanded our market offerings, developed unique assets, and grown significantly, with data engineering now comprising almost half of our company. With every success, I’ve progressed within the company, reflecting our shared growth and commitment to excellence.
What drew you to the field of data engineering, and how did you develop your expertise in this area?
My journey into data engineering began with a fascination for how systems communicate, likened to silent conversations. I started with simple SQL and as an early adopter of Hadoop in 2009-10, I honed skills in distributed computing and parallel data processing. The advent of cloud made data engineering more accessible, stoking my interest further.
While the technology evolved, core principles remained constant, enabling me to adapt quickly. Beyond technical skills, working across diverse domains like BFSI, healthcare, telecom, retail, and CPG enriched my expertise. I tackled complex problems, which, coupled with insights gained from a couple of industry pioneers, bolstered my confidence and technical acumen. In essence, my career has been a journey from databases to big data ecosystems and the cloud, constantly enhancing my proficiency in this dynamic field of data engineering.
Data science is a rapidly evolving field. How does Tredence stay ahead of the curve and ensure its solutions incorporate the latest advancements and best practices in the industry?
At Tredence, we constantly innovate to stay ahead in the rapidly evolving data science field. We have established an AI Center of Excellence, fueling our innovation flywheel with cutting-edge advancements.
We’ve built a Knowledge Management System that processes varied enterprise documents and includes a domain-specific Q&A system, akin to ChatGPT. We’ve developed a co-pilot integrated data science workbench, powered by GenAI algorithms and Composite AI, significantly improving our analysts’ productivity.
We’re also democratizing data insights for business users through our GenAI solution that converts Natural Language Queries into SQL queries, providing easy-to-understand insights. These are being implemented across our client environments, significantly adding value to their businesses.
How do you ensure data quality and integrity in your data engineering processes? What steps do you take to maintain data accuracy and consistency?
Data quality and integrity are fundamental to any organization, but they’re often mistakenly considered solely as technology issues. It’s crucial to remember that they’re also about people and processes. Empowering data and business stewards to own and maintain data assets is an integral part of the equation.
At Tredence, we’ve recognized this and developed D-Quest, our proprietary data quality framework. This solution handles data quality throughout the data and analytics lifecycle, from ingestion to processing to delivery. It allows the identification of critical data elements and the application of quality rules. These rules not only detect data quality issues, but also quarantine problematic records, with integrated alert systems to prompt corrective actions. While these rules may be heuristic-based, they’re further enhanced by AI and machine learning models.
Additionally, our focus lies on data observability – monitoring changes across the data landscape and creating accessible reports and dashboards to display data quality evolution. This transparency helps us measure the quality against critical data elements and track improvements or declines over time. In sum, our approach merges technology, people, and processes to ensure and continuously improve data quality.
Innovation is a key driver of technology thought leadership. How does Tredence foster a culture of innovation within the technology teams?
Innovation is at the heart of Tredence’s technology thought leadership. We identify potential innovation white spaces by closely studying business challenges our sales and customer success teams encounter, as well as the industry trends. Our centralized Studio team incubates accelerators under the ‘Horizon’ framework, depending on the relevance and immediacy of the identified challenges.
In addition, our technology-specific Centers of Excellence (COEs) continually develop new artifacts, technical know-how, and toolkits, particularly focusing on modern technology stacks such as Databricks and Snowflake. Our approach fosters a thriving culture of innovation within our technology teams, driving us to consistently create cutting-edge solutions for complex problems.
Can you discuss the company’s approach to data privacy, security, and compliance? How are these aspects incorporated into data engineering processes and systems?
At Tredence, we prioritize data privacy, security, and compliance as an integral part of our data engineering processes and systems. In our approach, we’re fully compliant with international regulations such as CCPA and GDPR, ensuring the highest standards of data protection.
Moreover, our data handling procedures safeguard both data at rest and in transit, employing custom-made encryption frameworks for optimal security. By ingraining these practices into our systems, we not only uphold our commitment to data privacy and protection but also foster a culture of trust and reliability with our clients and stakeholders.
In your experience, what are the primary sources of big data, and how can organizations effectively manage and leverage these diverse data sources?
Primary sources of big data for organizations include operational systems, CRM databases, IoT devices, social media interactions, and more. To manage this data effectively, organizations need robust data infrastructure backed by distributed computing and cloud technologies. Leveraging these sources necessitates the adoption of advanced analytics, machine learning, and AI.
Tredence’s GenAI solutions, such as the co-pilot integrated data science workbench, can significantly enhance data processing and insights. Ensuring data privacy and security through GDPR and CCPA compliant practices, and robust encryption, is also crucial. Ultimately, the real value of big data lies in the actionable insights extracted, driving business growth and innovation.
Fine-grained access control to relevant consumers on certified data assets and the ability to glean insights from those.
How does Tredence leverage data science to address specific challenges faced by businesses and industries?
Tredence, as a specialized AI and technology firm, delivers bespoke solutions tailored to businesses’ unique needs, leveraging cutting-edge data science concepts and methodologies. Our accelerator-led approach significantly enhances time to value, surpassing traditional consulting and technology companies by more than 50%. Tredence offers a comprehensive suite of services that cover the entire AI/ML value chain, supporting businesses at every stage of their data science journey.
Our Data Science services empower clients to seamlessly progress from ideation to actionable insights, enabling ML-driven data analytics and automation at scale and velocity. Tredence’s solutioning services span critical domains such as Pricing & Promotion, Supply Chain Management, Marketing Science, People Science, Product Innovation, Digital Analytics, Fraud Mitigation, Loyalty Science, and Customer Lifecycle Management.
Focusing on advanced data science frameworks, Tredence excels in developing sophisticated Forecasting, NLP models, Optimization Engines, Recommender systems, Image and video processing algorithms, Generative AI Systems, Data drift detection, and Model explainability techniques. This comprehensive approach enables businesses to harness the full potential of data science, facilitating well-informed decision-making and driving operational efficiency and growth across various business functions. By incorporating these data science concepts into their solutions, Tredence empowers businesses to gain a competitive advantage and capitalize on data-driven insights.
Building and scaling high-performance technology teams is a crucial aspect of leadership. Can you describe your approach to building strong technology teams?
At Tredence, we believe in fostering high-performance technology teams through personalized, adaptive learning.
We dedicate 20% of our team’s time to training and development via various programs. From sharing stories of success in our WALL (Weekly Agenda for Lots of Learning) program to imparting technical skills in the Full Stack Analyst Program and DE Certifications, we aim to enhance our team’s skills holistically.
We sponsor access to Harvard Manage Mentor for leadership skills, while programs like Everest and ASHTA further strengthen these capabilities. Our Career Scout program encourages individual growth, and the TALL (Tredence Academy for Lots of Learning) program broadens generalist skills in DataOps, MLOps and DevOps. Lastly, our ‘U Learn V Pay’ initiative promotes cross-skilling and upskilling.
Looking ahead, what do you envision as the future of data engineering? How do you see the role evolving in the coming years?
In the future of data engineering, three trends stand out.
First, a shift from foundational data frameworks to application of data, emphasizing the simplification of AI and ML models and democratization of these tools via natural language queries.
Second, quantum computing, promising increased data processing speed and cost-efficiency, will significantly alter the landscape. Lastly, new gen AI coding paradigms will drastically reduce time-to-value. In sum, the future of data engineering lies in embracing AI/ML, capitalizing on quantum computing, and democratizing these technologies, ultimately redefining the role of data engineers.
Arnab Sen
VP-Data Engineering at Tredence Inc
Arnab Sen is an experienced professional with a career spanning over 16 years in the technology and decision science industry. He presently serves as the VP-Data Engineering at Tredence, a prominent data analytics company, where he helps organizations design their AI-ML/Cloud/Big-data strategies. With his expertise in data monetization, Arnab uncovers the latent potential of data to drive business transformations across B2B & B2C clients from diverse industries.
