Accelerated Computing in Modern Data Centers With Datapelago

Data Engineering Podcast

Player FM - Internet Radio Done Right

68 subscribers

Aggiunto due anni fa

Contenuto fornito da Tobias Macey. Tutti i contenuti dei podcast, inclusi episodi, grafica e descrizioni dei podcast, vengono caricati e forniti direttamente da Tobias Macey o dal partner della piattaforma podcast. Se ritieni che qualcuno stia utilizzando la tua opera protetta da copyright senza la tua autorizzazione, puoi seguire la procedura descritta qui https://it.player.fm/legal.

State Secrets: Inside The Making Of The Electric State

1
The Secret To Getting Inspired: Millie Bobby Brown & Chris Pratt Go Behind The Scenes 21:04

7 weeks fa21:04

Riproduci in seguito

Liste

Like aggiunto

21:04

Step into the mysterious and visually stunning world of The Electric State as host Francesca Amiker takes you behind the scenes with the creative masterminds who brought Simon Stålenhag’s dystopian vision to life. In this premiere episode, directors Joe and Anthony Russo, stars Millie Bobby Brown and Chris Pratt, writers Christopher Markus and Stephen McFeely, and producers Angela Russo-Otstot and Chris Castaldi reveal how they transformed a haunting graphic novel into an epic cinematic experience. Watch The Electric State coming to Netflix on March 14th. Check out more from Netflix Podcasts . State Secrets: Inside the Making of The Electric State is produced by Netflix and Treefort Media.…

circa un anno fa 55:36

MP3•Pagina principale dell'episodio

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
Your host is Tobias Macey and today I'm interviewing Rajan Goyal about how to drastically improve efficiencies in data processing by re-imagining the system architecture

Interview

Introduction
How did you get involved in the area of data management?
Can you start by outlining the main factors that contribute to performance challenges in data lake environments?
The different components of open data processing systems have evolved from different starting points with different objectives. In your experience, how has that un-planned and un-synchronized evolution of the ecosystem hindered the capabilities and adoption of open technologies?
The introduction of a new cross-cutting capability (e.g. Iceberg) has typically taken a substantial amount of time to gain support across different engines and ecosystems. What do you see as the point of highest leverage to improve the capabilities of the entire stack with the least amount of co-ordination?
What was the motivating insight that led you to invest in the technology that powers Datapelago?
Can you describe the system design of Datapelago and how it integrates with existing data engines?
The growth in the generation and application of unstructured data is a notable shift in the work being done by data teams. What are the areas of overlap in the fundamental nature of data (whether structured, semi-structured, or unstructured) that you are able to exploit to bridge the processing gap?
What are the most interesting, innovative, or unexpected ways that you have seen Datapelago used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Datapelago?
When is Datapelago the wrong choice?
What do you have planned for the future of Datapelago?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

462 episodi

#Tech #Tobias Macey #Data Engineering #Big Data #Database #Data Pipeline #Podcasting Education

Accelerated Computing in Modern Data Centers With Datapelago

Data Engineering Podcast

68 subscribers

published circa un anno fa

MP3•Pagina principale dell'episodio

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
Your host is Tobias Macey and today I'm interviewing Rajan Goyal about how to drastically improve efficiencies in data processing by re-imagining the system architecture

Interview

Introduction
How did you get involved in the area of data management?
Can you start by outlining the main factors that contribute to performance challenges in data lake environments?
The different components of open data processing systems have evolved from different starting points with different objectives. In your experience, how has that un-planned and un-synchronized evolution of the ecosystem hindered the capabilities and adoption of open technologies?
The introduction of a new cross-cutting capability (e.g. Iceberg) has typically taken a substantial amount of time to gain support across different engines and ecosystems. What do you see as the point of highest leverage to improve the capabilities of the entire stack with the least amount of co-ordination?
What was the motivating insight that led you to invest in the technology that powers Datapelago?
Can you describe the system design of Datapelago and how it integrates with existing data engines?
The growth in the generation and application of unstructured data is a notable shift in the work being done by data teams. What are the areas of overlap in the fundamental nature of data (whether structured, semi-structured, or unstructured) that you are able to exploit to bridge the processing gap?
What are the most interesting, innovative, or unexpected ways that you have seen Datapelago used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Datapelago?
When is Datapelago the wrong choice?
What do you have planned for the future of Datapelago?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

462 episodi

#Tech #Tobias Macey #Data Engineering #Big Data #Database #Data Pipeline #Podcasting Education

Tutti gli episodi

Summary In this episode of the Data Engineering Podcast Sam Kleinman talks about the pivotal role of databases in software engineering. Sam shares his journey into the world of data and discusses the complexities of database selection, highlighting the trade-offs between different database architectures and how these choices affect system design, query performance, and the need for ETL processes. He emphasizes the importance of understanding specific requirements to choose the right database engine and warns against over-engineering solutions that can lead to increased complexity. Sam also touches on the tendency of engineers to move logic to the application layer due to skepticism about database longevity and advises teams to leverage database capabilities instead. Finally, he identifies a significant gap in data management tooling: the lack of easy-to-use testing tools for database interactions, highlighting the need for better testing paradigms to ensure reliability and reduce bugs in data-driven applications. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management It’s 2024, why are we still doing data migrations by hand? Teams spend months—sometimes years—manually converting queries and validating data, burning resources and crushing morale. Datafold's AI-powered Migration Agent brings migrations into the modern era. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today to learn how Datafold can automate your migration and ensure source to target parity. Your host is Tobias Macey and today I'm interviewing Sam Kleinman about database tradeoffs across operating environments and axes of scale Interview Introduction How did you get involved in the area of data management? The database engine you use has a substantial impact on how you architect your overall system. When starting a greenfield project, what do you see as the most important factor to consider when selecting a database? points of friction introduced by database capabilities embedded databases (e.g. SQLite, DuckDB, LanceDB), when to use and when do they become a bottleneck single-node database engines (e.g. Postgres, MySQL), when are they legitimately a problem distributed databases (e.g. CockroachDB, PlanetScale, MongoDB) polyglot storage vs. general-purpose/multimodal databases federated queries, benefits and limitations ease of integration vs. variability of performance and access control Contact Info LinkedIn GitHub Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story. Links MongoDB Neon Podcast Episode GlareDB NoSQL S3 Conditional Write Event driven architecture CockroachDB Couchbase Cassandra The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA…

1
Bridging Code and UI in Data Orchestration with Kestra 44:30

21 weeks fa44:30

Riproduci in seguito

Liste

Like aggiunto

44:30

Summary In this episode of the Data Engineering Podcast, Anna Geller talks about the integration of code and UI-driven interfaces for data orchestration. Anna defines data orchestration as automating the coordination of workflow nodes that interact with data across various business functions, discussing how it goes beyond ETL and analytics to enable real-time data processing across different internal systems. She explores the challenges of using existing scheduling tools for data-specific workflows, highlighting limitations and anti-patterns, and discusses Kestra's solution, a low-code orchestration platform that combines code-driven flexibility with UI-driven simplicity. Anna delves into Kestra's architectural design, API-first approach, and pluggable infrastructure, and shares insights on balancing UI and code-driven workflows, the challenges of open-core business models, and innovative user applications of Kestra's platform. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details. As a listener of the Data Engineering Podcast you clearly care about data and how it affects your organization and the world. For even more perspective on the ways that data impacts everything around us you should listen to Data Citizens® Dialogues, the forward-thinking podcast from the folks at Collibra. You'll get further insights from industry leaders, innovators, and executives in the world's largest companies on the topics that are top of mind for everyone. They address questions around AI governance, data sharing, and working at global scale. In particular I appreciate the ability to hear about the challenges that enterprise scale businesses are tackling in this fast-moving field. While data is shaping our world, Data Citizens Dialogues is shaping the conversation. Subscribe to Data Citizens Dialogues on Apple, Spotify, Youtube, or wherever you get your podcasts. Your host is Tobias Macey and today I'm interviewing Anna Geller about incorporating both code and UI driven interfaces for data orchestration Interview Introduction How did you get involved in the area of data management? Can you start by sharing a definition of what constitutes "data orchestration"? There are many orchestration and scheduling systems that exist in other contexts (e.g. CI/CD systems, Kubernetes, etc.). Those are often adapted to data workflows because they already exist in the organizational context. What are the anti-patterns and limitations that approach introduces in data workflows? What are the problems that exist in the opposite direction of using data orchestrators for CI/CD, etc.? Data orchestrators have been around for decades, with many different generations and opinions about how and by whom they are used. What do you see as the main motivation for UI vs. code-driven workflows? What are the benefits of combining code-driven and UI-driven capabilities in a single orchestrator? What constraints does it necessitate to allow for interoperability between those modalities? Data Orchestrators need to integrate with many external systems. How does Kestra approach building integrations and ensure governance for all their underlying configurations? Managing workflows at scale across teams can be challenging in terms of providing structure and visibility of dependencies across workflows and teams. What features does Kestra offer so that all pipelines and teams stay organised? What are the most interesting, innovative, or unexpected ways that you have seen Kestra used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on Kestra? When is Kestra the wrong choice? What do you have planned for the future of Kestra? Contact Info LinkedIn Blog Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story. Links Kestra CI/CD State Machine AWS Lambda GitHub Actions ECS Fargate Airflow Kafka Elasticsearch Airflow XCom The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA In this episode of the Data Engineering Podcast, host Tobias Macy interviews Anna Geller, a data engineer turned product manager, about the integration of code and UI-driven interfaces for data orchestration. Anna shares her journey from working with data during an internship at KPMG to her current role as a product lead at Kestra. She provides her insights into the concept of data orchestration, emphasizing its broader scope beyond just ETL and analytics, and discusses the challenges and anti-patterns that arise when using existing scheduling systems for data-specific workflows. Anna explains the overlap between CI/CD, scheduling, and orchestration tools, and the limitations that occur when these tools are used for data workflows. She highlights the importance of visibility and governance at scale and the need for a dedicated orchestrator like Kestra. The conversation also delves into the challenges of using data orchestrators for non-data workflows and the benefits of combining code and UI-driven approaches. Anna discusses Kestra's architecture, which supports both JDBC and Kafka backends, and its focus on API-first interactions. She explains how Kestra handles task granularity, inputs, and outputs, and the flexibility provided by its plugin system. The episode also explores Kestra's approach to data as assets, the target audience for Kestra, and how it bridges different workflows across organizational boundaries. The discussion touches on Kestra's open-core model, the challenges of balancing open-source and enterprise features, and the innovative ways Kestra is being applied. Anna shares insights into Kestra's local development experience, the lessons learned in building the product, and the upcoming features and projects that Kestra is excited to explore.…

Benvenuto su Player FM!

Player FM ricerca sul web podcast di alta qualità che tu possa goderti adesso. È la migliore app di podcast e funziona su Android, iPhone e web. Registrati per sincronizzare le iscrizioni su tutti i tuoi dispositivi.

Ascolta più di 500 argomenti

68 subscribers

Simile a Data Engineering Podcast

The Let Them Theory: A Life-Changing Tool That Millions of People Can't Stop Talking About

Mini Mic Pro (Newest Model), Wireless Microphone for iPhone, iPad, Android, Lavalier Microphone for Video Recording - 2 Pack iPhone Mic Crystal Clear Recording with USB-C for Podcast, ASMR

Bounty Quick Size Paper Towels, White, 8 Family Rolls = 20 Regular Rolls (Packaging May Vary)

Podcast che vale la pena ascoltare

Data Engineering Podcast « » Accelerated Computing in Modern Data Centers With Datapelago

Accelerated Computing in Modern Data Centers With Datapelago

Podcast che vale la pena ascoltare

Benvenuto su Player FM!

The Let Them Theory: A Life-Changing Tool That Millions of People Can't Stop Talking About

The Let Them Theory: A Life-Changing Tool That Millions of People Can't Stop Talking About

Amazon Basics Multipurpose Copy Printer Paper, 8.5" x 11", 20 lb, 8 Reams, 4000 Sheets, 92 Bright, White

Ailun 3 Pack Screen Protector for iPhone 16 Pro Max [6.9 inch] + 3 Pack Camera Lens Protector with Installation Frame,Sensor Protection,Dynamic Island Compatible,Case Friendly Tempered Glass Film

The Let Them Theory: A Life-Changing Tool That Millions of People Can't Stop Talking About

Simile a Data Engineering Podcast

Guida rapida

Data Engineering Podcast « »
Accelerated Computing in Modern Data Centers With Datapelago