I Used Microsoft Copilot for Fabric and Saved Hours—Here’s How

From Code Cruncher to Creative Thinker: How Microsoft Copilot in Fabric Rewired My Data Engineering JourneyEver spent what felt like an entire summer afternoon just transforming a CSV file? I have—and to say it sapped my motivation would be an understatement. But that was before Microsoft Copilot entered the chat. In this post, I’ll share the winding, sometimes embarrassing, sometimes revelatory path I took from dreading routine data engineering work to rediscovering why I loved building things with code in the first place—all thanks to a little AI magic (and a few hard-learned lessons).When Burnout Met Automation: A Cautionary TaleI used to lose entire weekends to CSV file conversions. Not kidding. My Saturdays would dissolve into a blur of error messages while debugging Spark code that refused to cooperate. Coffee cups would pile up as the sun went down, and I’d realize another day had vanished into the digital void.Sound familiar?The Weekend-Eating MonsterConverting files from CSV to Delta Parquet tables was my personal nemesis. What should have been simple became a soul-crushing time sink. I’d start Friday evening thinking, “This’ll take an hour, tops.” By Sunday night, I’d be questioning my career choices.Research backs up my pain – automation can reduce routine task times by up to 40%. But knowing that didn’t help when I was knee-deep in code errors.Skepticism: My Default SettingWhen Copilot promised to handle these tasks, I laughed. Seriously? Hand over my code to an AI assistant? The trust issues were real.* What if it made mistakes I wouldn’t catch?* What if it created more problems than solutions?* What if I became… replaceable?But desperation eventually trumped skepticism.Old Me vs. New MeThe transformation was almost embarrassing:Old me: Spent 6+ hours creating a fiscal calendar, cursing at my screen.New me: Types a prompt, reviews the generated code, done in 15 minutes.Manual data transformation tasks that once devoured my weekends now take minutes. ETL workflows that used to require days of coding and debugging? Handled through natural language prompts.”Sometimes, freeing yourself from a tedious workflow is the most creative thing you can do.” – Inder RanaRana’s words hit different now. The relief of letting go was unexpected. I found myself having actual free time. I rediscovered hobbies. I remembered what my family looked like.The Surprising AftermathThe biggest shock wasn’t the efficiency gain – it was the mental space that opened up. Without the dread of endless debugging sessions, my mind wandered to bigger questions and creative solutions.Yes, I still review everything Copilot generates. Yes, I sometimes need to tweak the code. But the 40% time savings? In my case, that’s a conservative estimate.My burnout didn’t just meet automation. It was thoroughly defeated by it.The Lost Art of Prompt Engineering (Or: Talking To Robots For Fun And Profit)I never thought I’d develop a creative relationship with an AI, but here we are. Writing prompts for Copilot has somehow become one of the most unexpectedly creative parts of my job as a data engineer.Remember when programming meant memorizing exact syntax? Those days feel distant now.The Accidental Monster FactoryLast month, I was exhausted after a long day of data wrangling. My brain was fried. I needed to create a simple data transformation table, but somehow typed: “create fantasy monster table with damage stats and special abilities.”Copilot’s response? A bizarre mix of SQL syntax and fantasy RPG content that made absolutely no sense. It tried to create columns for “acidBreath” and “tentacleCount” alongside my actual data fields.I laughed for five minutes straight. Then realized something important: I was talking to my development environment. Not coding. Talking.The Prompt-Review-Improve LoopI’ve developed a workflow now:* Write a natural language prompt* Review what Copilot generates* Refine my prompt with more details* Repeat until perfectIt’s less like programming and more like… coaching? Directing? Whatever it is, it’s changing how I approach problems.Learning From The ProsIndustry demos have been eye-opening. Inder Rana showed how Copilot could read files from CMS prescription folders into Spark data frames with just conversational prompts.Dan Taylor’s demo converting Azure SQL data into date tables blew my mind. As he said,”The art of prompt engineering is the new craft for data engineers.”I’m starting to believe him.Getting ComplexMy prompts have evolved beyond simple tasks. Now I’m asking for column conversions, data type transformations, and even new calculated columns based on business logic.Sometimes my requests go sideways—I once got a perfect poetry analysis instead of database code because I wasn’t specific enough. But that’s part of the learning curve.This new interface—natural language—feels more intuitive than traditional scripting ever did. It’s not perfect. You need human oversight. But I’m spending more time thinking about what I want to accomplish rather than how to accomplish it.And honestly? That feels like progress.ETL in Plain English: Goodbye Cryptic ScriptsRemember the old days of ETL? I sure do. A mess of scripts sprawled across multiple files, confusing data type conversions, and those dreaded broken data pipes that would bring everything crashing down at 2 AM. Good times… not.From Chaos to ConversationNow? I literally just describe what I want to Copilot:”Pull last quarter’s sales data from our SQL database, clean up the null values in the customer_id field, and create a summary table with regional totals.”And just like that, Copilot assembles the code on the fly. No more hunting through Stack Overflow or deciphering cryptic documentation. It’s almost unfair how simple it’s become.Magic Commands That Feel Like CheatingThe chart magic commands? Pure wizardry. Instead of spending hours tweaking visualization code, I just type something like %%create_chart sales by region and boom—instant visualization.And don’t get me started on %%fix_errors in notebooks. That command has saved me countless debugging hours. It feels like having a senior developer looking over my shoulder, catching mistakes before they cause problems.When Copilot Sees What You Don’tLast week, I was transforming some customer data when Copilot politely suggested: “I notice you’re trying to join these tables on different column types. Would you like me to add a conversion step?”I hadn’t even spotted the issue! That would have been hours of debugging down the drain.Trust, But VerifyIs every Copilot suggestion perfect? Nope. Sometimes it generates code that looks plausible but doesn’t quite work for my specific scenario. But here’s what I’ve noticed: the mistakes are becoming fewer, and I’m getting better at prompting it correctly.* The tedious parts of ETL now feel almost playful* My focus has shifted from fixing code to designing workflows* Human review is still essential, but much less painfulAs Josh de put it: “With Copilot, describing data flows in plain English isn’t just possible—it’s liberating.”I’m not throwing away my coding skills anytime soon. But I am embracing a new reality where ETL creation has transformed from slow and tedious to fast and, dare I say, enjoyable. And that’s something worth celebrating.From Days to Minutes: Fiscal Calendars Without the FussI still get that sinking feeling when I think about fiscal calendar projects. You know the ones—tedious, time-consuming table creation that somehow always lands on your desk.For years, I’d block out entire afternoons (sometimes days) to build these calendars from scratch. Coding each parameter, double-checking date ranges, fixing the inevitable bugs. It was… painful.The Game-Changer ApproachThen I saw Greg Bowmont’s demonstration. My jaw literally dropped.He showed how Copilot could generate custom fiscal date calendars almost instantly. Not in days. Not in hours. In minutes.”Automating the fiscal calendar put hours back into my quarter. That’s ROI you can feel.” – Greg BowmontWhat used to consume half my week now takes less time than my coffee break. That’s not an exaggeration—I timed it!The Secret Sauce: Configurable Parameters* Column specifications tailored to your needs* Flexible data types (no more conversion headaches)* Custom date ranges that align with any fiscal structureThese configurable parameters change everything. Instead of building from zero, I simply tell Copilot what I need, and it generates the base code instantly.A Wild ThoughtImagine a world where finance teams build their own fiscal calendars without ever opening a code editor. Where they don’t need to wait for IT or data engineering to find time in their sprint.We’re surprisingly close to that reality. The finance director in my company—who has zero coding experience—recently used my Copilot prompt template to generate a custom calendar for a special project.The Human Touch Still MattersI’m not saying it’s perfect right out of the box. A quick review is still necessary—tweaking date formats here, adjusting column names there. Sometimes business-specific calculations need adding.But starting with 90% of the work done? That’s a game-changer.When I think about all those days I spent hunched over fiscal tables… well, I wish I could get those hours back. At least now, with Copilot generating the heavy lifting, I can focus on the interesting parts of data engineering instead.Lost in Legacy Code? Copilot as Decoder RingWe’ve all been there. That dreaded legacy codebase nobody wants to touch. The one with sparse documentation and cryptic variable names that make you question your career choices.Last month, I inherited “the beast” – a 15,000-line monstrosity written by a developer who left three years ago. My stomach dropped when my manager cheerfully assigned it to me.The Legacy Code NightmareNormally, I’d spend days just trying to understand what the code actually did, let alone fix the reported bugs. But this ti

Become a supporter of this podcast: https://www.spreaker.com/podcast/m365-fm-modern-work-security-and-productivity-with-microsoft-365–6704921/support.

If this clashes with how you’ve seen it play out, I’m always curious. I use LinkedIn for the back-and-forth.