"Stay Focused, Go After Your Dreams and Keep Moving Forward" - LL Cool J But Remember to Chase 1 Dream at a Time, Not Multiple Dreams at the same time. There is a Time and Place for Everything But Not Everything at the Same Time. Imagine Every Dream is a Race Track by itself, and that there is prize and momentum to be won at the end of each race. How can you finish even one race if you keep hopping onto different race tracks? And by when would you get the reward? Even if You Failed or Suffered a Setback, Fail Forward, Fail Faster to Recover and Resume Faster. #StayFocused #Focus #Focused #FocusedOnMyGoals #FocusOnYourself #OneGoalAtATime #GoForYourDreams #Prioritize #Priorities #Priority #FirstThingsFirst #Failure #Fail #Failing #FailingForward #FailForward #FailFaster #MultiTasking #EyesOnThePrize #OneTrackMind #Faster #Dreams #MovingForward #KeepMovingForward #KeepGoing #KeepGoingForward #Forward #NeverGiveUp https://www.instagram.com/p/Cfg-C7HpQA8/?igshid=NGJjMDIxMWI=

[Fabric] Entre Archivos y Tablas de Lakehouse - SQL Notebooks

Ya conocemos un panorama de Fabric y por donde empezar. La Data Web nos mostró unos artículos sobre esto. Mientras más veo Fabric más sorprendido estoy sobre la capacidad SaaS y low code que generaron para todas sus estapas de proyecto.

Un ejemplo sobre la sencillez fue copiar datos con Data Factory. En este artículo veremos otro ejemplo para que fanáticos de SQL puedan trabajar en ingeniería de datos o modelado dimensional desde un notebook.

Arquitectura Medallón

Si nunca escuchaste hablar de ella te sugiero que pronto leas. La arquitectura es una metodología que describe una capas de datos que denotan la calidad de los datos almacenados en el lakehouse. Las capas son carpetas jerárquicas que nos permiten determinar un orden en el ciclo de vida del datos y su proceso de transformaciones.

Los términos bronce (sin procesar), plata (validado) y oro (enriquecido/agrupado) describen la calidad de los datos en cada una de estas capas.

Ésta metodología es una referencia o modo de trabajo que puede tener sus variaciones dependiendo del negocio. Por ejemplo, en un escenario sencillo de pocos datos, probablemente no usaríamos gold, sino que luego de dejar validados los datos en silver podríamos construir el modelado dimensional directamente en el paso a "Tablas" de Lakehouse de Fabric.

NOTAS: Recordemos que "Tablas" en Lakehouse es un spark catalog también conocido como Metastore que esta directamente vinculado con SQL Endpoint y un PowerBi Dataset que viene por defecto.

¿Qué son los notebooks de Fabric?

Microsoft los define como: "un elemento de código principal para desarrollar trabajos de Apache Spark y experimentos de aprendizaje automático, es una superficie interactiva basada en web que usan los científicos de datos e ingenieros de datos para escribir un código que se beneficie de visualizaciones enriquecidas y texto en Markdown."

Dicho de manera más sencilla, es un espacio que nos permite ejecutar bloques de código spark que puede ser automatizado. Hoy por hoy es una de las formas más populares para hacer transformaciones y limpieza de datos.

Luego de crear un notebook (dentro de servicio data engineering o data science) podemos abrir en el panel izquierdo un Lakehouse para tener referencia de la estructura en la cual estamos trabajando y el tipo de Spark deseado.

Spark

Spark se ha convertido en el indiscutible lenguaje de lectura de datos en un lake. Así como SQL lo fue por años sobre un motor de base de datos, ahora Spark lo es para Lakehouse. Lo bueno de spark es que permite usar más de un lenguaje según nuestro comodidad.

Creo que es inegable que python está ocupando un lugar privilegiado junto a SQL que ha ganado suficiente popularidad como para encontrarse con ingenieros de datos que no conocen SQL pero son increíbles desarrolladores en python. En este artículo quiero enfocarlo en SQL puesto que lo más frecuente de uso es Python y podríamos charlar de SQL para aportar a perfiles más antiguos como DBAs o Data Analysts que trabajaron con herramientas de diseño y Bases de Datos.

Lectura de archivos de Lakehouse con SQL

Lo primero que debemos saber es que para trabajar en comodidad con notebooks, creamos tablas temporales que nacen de un esquema especificado al momento de leer la información. Para el ejemplo veremos dos escenarios, una tabla Customers con un archivo parquet y una tabla Orders que fue particionada por año en distintos archivos parquet según el año.

CREATE OR REPLACE TEMPORARY VIEW Dim_Customers_Temp USING PARQUET OPTIONS ( path "Files/Silver/Customers/*.parquet", header "true", mode "FAILFAST" ) ;

CREATE OR REPLACE TEMPORARY VIEW Orders USING PARQUET OPTIONS ( path "Files/Silver/Orders/Year=*", header "true", mode "FAILFAST" ) ;

Vean como delimitamos la tabla temporal, especificando el formato parquet y una dirección super sencilla de Files. El "*" nos ayuda a leer todos los archivos de una carpeta o inclusive parte del nombre de las carpetas que componen los archivos. Para el caso de orders tengo carpetas "Year=1998" que podemos leerlas juntas reemplazando el año por asterisco. Finalmente, especificamos que tenga cabeceras y falle rápido en caso de un problema.

Consultas y transformaciones

Una vez creada la tabla temporal, podremos ejecutar en la celda de un notebook una consulta como si estuvieramos en un motor de nuestra comodidad como DBeaver.

Escritura de tablas temporales a las Tablas de Lakehouse

Realizadas las transformaciones, joins y lo que fuera necesario para construir nuestro modelado dimensional, hechos y dimensiones, pasaremos a almacenarlo en "Tablas".

Las transformaciones pueden irse guardando en otras tablas temporales o podemos almacenar el resultado de la consulta directamente sobre Tablas. Por ejemplo, queremos crear una tabla de hechos Orders a partir de Orders y Order details:

CREATE TABLE Fact_Orders USING delta AS SELECT od.*, o.CustomerID, o.EmployeeID, o.OrderDate, o.Freight, o.ShipName FROM OrdersDetails od LEFT JOIN Orders o ON od.OrderID = o.OrderID

Al realizar el Create Table estamos oficialmente almacenando sobre el Spark Catalog. Fíjense el tipo de almacenamiento porque es muy importante que este en DELTA para mejor funcionamiento puesto que es nativo para Fabric.

Resultado

Si nuestro proceso fue correcto, veremos la tabla en la carpeta Tables con una flechita hacia arriba sobre la tabla. Esto significa que la tabla es Delta y todo está en orden. Si hubieramos tenido una complicación, se crearía una carpeta "Undefinied" en Tables la cual impide la lectura de nuevas tablas y transformaciones por SQL Endpoint y Dataset. Mucho cuidado y siempre revisar que todo quede en orden:

Pensamientos

Así llegamos al final del recorrido donde podemos apreciar lo sencillo que es leer, transformar y almacenar nuestros modelos dimensionales con SQL usando Notebooks en Fabric. Cabe aclarar que es un simple ejemplo sin actualizaciones incrementales pero si con lectura de particiones de tiempo ya creadas por un data engineering en capa Silver.

¿Qué hay de Databricks?

Podemos usar libremente databricks para todo lo que sean notebooks y procesamiento tal cual lo venimos usando. Lo que no tendríamos al trabajar de ésta manera sería la sencillez para leer y escribir tablas sin tener que especificar todo el ABFS y la característica de Data Wrangler. Dependerá del poder de procesamiento que necesitamos para ejecutar el notebooks si nos alcanza con el de Fabric o necesitamos algo particular de mayor potencia. Para más información pueden leer esto: https://learn.microsoft.com/en-us/fabric/onelake/onelake-azure-databricks

Espero que esto los ayude a introducirse en la construcción de modelados dimensionales con clásico SQL en un Lakehouse como alternativa al tradicional Warehouse usando Fabric. Pueden encontrar el notebook completo en mi github que incluye correr una celda en otro lenguaje y construcción de tabla fecha en notebook.

From a W logo drawn on a napkin to a nationwide sold brand ✨🛡️ Warrior Beverages has taken the business world by storm! 💪💼 Fail fast, fail forward, and watch your progress soar 🚀 Don’t wait, find us on Amazon and warriorbeverages.com today! 🌟

#WarriorBeverages #SuccessStory #Nationwide #AmazonFinds #BusinessGoals #FailFast #FailForward #ProgressMatters #RiseToTheTop #WarriorSpirit #AmazonShopping #DrinkWarrior #BeYourOwnWarrior #BusinessMindset #ShopSmallBusiness #cpgbrand #brand #podcast #news #media

In fact, my human voice appreciates my idiosyncrasies, the imperfect nuances of myself and my relationships. It loves the parts of me that might seem “messy,” because it understands that humans are inherently imperfect. It understands that I can be messy and worthy.

Elisha Mudly, a South African born writer/actor/yogi/creator living in New York City.

If you fall flat on your face, at least you’re moving forward. All you have to do is get back up and try again.

Richard Brandon

Tip 9 today if you are a male entrepreneur consider starting a business with a woman. Statistics show start up businesses involving women, are more likely to survive. #bouncepreneurs #bouncebackinbusiness #startup #startups #entrepreneur #entrepreneurlife #womenentrepreneurs #femaleentrepreneur #business #innovation #startuplife #failfast #startuptips #startupquotes #startupstory#startupideas #entrepreneure #entrepreneure #soletrader #soletraders #businesswales https://www.instagram.com/p/Bz7lr6-nNbF/?igshid=j0r8ifjcjb2g

Still can’t get the hang on water colors. #failfast #practice #watercolor #character https://www.instagram.com/p/Br01gw8DXS5/?utm_source=ig_tumblr_share&igshid=1nhybsn0yka3o

𝐓𝐞𝐬𝐭 𝐀𝐮𝐭𝐨𝐦𝐚𝐭𝐢𝐨𝐧 𝐒𝐭𝐫𝐚𝐭𝐞𝐠𝐲 𝐰𝐢𝐭𝐡 𝐅𝐚𝐢𝐥 𝐅𝐚𝐬𝐭 𝐀𝐩𝐩𝐫𝐨𝐚𝐜𝐡 #testautomation #automation #testing #livewebinars Failing fast is not the answer to every business problem that exists in the world, but it is a useful methodology for managing uncertainty and risk in an intelligent, cost-effective way and it also helps you assess the best way to solve it. Join us now to understand how to integrate fail fast approach in automation? https://www.events.sailotech.com/test-automation-strategy-with-fail-fast-approach/

Everyday items - this image may not look like much to you, but to me it looks like excellence. It is a tube of concealer made with avocado cream and no harmful ingredients, a broken concealer applicator, and a new replacement applicator. How do everyday items, broken and replaced translate to excellence? By being honest, transparent, accommodating, and by failing and trying harder. I am a @beautycounter fan girl and Consultant and our Creamy Concealer has had a defective packaging issue where the doe foot has broken on many applicators. Beautycounter has decided to stop selling this product to redesign the wand and that process will not be complete until NEXT SPRING. This product itself is incredible (my favorite BC cosmetic in fact!) and the stop sell is in no way related to the quality of the product. Anyone in need of Concealer in the next few months or who already knows and loves this product should stock up by this MONDAY AT 11PM. If you have had a broken applicator, regardless of how long ago, we may be able to issue refunds or replacements beyond our 60 day return policy, so please let me know as soon as possible. Beautycounter’s dedication to excellence is exactly why we are pulling this product. This will be expensive and messy for the brand, but it is such an honor to witness a large, scaled company do the right thing for its Clients! We have free shipping on $50+ orders for Clients this month and anyone new to Beautycounter can save 20% on their first order with code CLEANFORALL20! My hope (and expectation) is that Beautycounter will continue to replace and refund defective Concealers (at least offering replacement wands 🤞🏽) throughout this process. I’m stocking up and am happy to help shade match and make recommendations! We still have fabulous foundation options and a makeup collection that comes with (the concealer till Monday night) a built in savings and full set of everything you need for a flawless finish! It’s called Flawless in Five and (till Monday at 11PM EST) comes with Concealer, Foundation, mascara, brow pencil/gel, blush and lip gloss for $150! 🪞 🧖🏽‍♀️ 💋 . #bcworks #failfast #betterbeauty #brandsthatwalkthewalk #cleanercosmetics (at Themis and Thread) https://www.instagram.com/p/CU2mFXNgt7G/?utm_medium=tumblr

Where’s my weekly shipment pic?!?! 🧐 So I don’t really have one this week... 😅 Because I did a thing... 😳 I stepped out of my comfort zone.... and here we are waiting for that to process. 😅 🗣 Y’all! I bit the bullet and placed an order with a new distributor. This will be my largest order since my first Amazon bust of a purchase. 🥴 I’m sure I don’t have to explain why it could be a little nerve-wracking today 🥸 But at least this time around I can rest assured knowing that I did learn from last time so I wasn’t just shooting in the dark! (You knew the bullet points were coming haha) 🦝 I know everything in my pallet already... because I selected it personally. I know it’s BSR, who else is on the listing, plus how many they have in current stock. In a nutshell: The entire pallet is data driven! 👀 🦝 I understand Amazon parameters better now than I did back then. Trial and error is the best teacher... oh, and my amazing sellers group! 🤓 🦝 Plus, I NOW know, if not Amazon, there is a platform somewhere for it! 🤪 Had I known that initially, I would have pulled all of my slow moving products from Amazon prior to hitting LTS fees - during one of their biannual inventory warehouse clean outs - and listed them all on posh, eBay, merc... which IS WHERE THE LAST STRAGGLERS ENDED UP SELLING! 🤪 I spent Monday from noon on to 4 AM researching products from their catalog and ended up filling a pallet. 😬😅😆 I spent the next day visiting a friend and then had a date with my mister. 🥰 That’s like the best part of being an entrepreneur. I worked when most slept so I had free time when most worked. (Even if work never really stops when you’re an entrepreneur ha) Okok, I’m rambling now but... YALL.... I don’t even know where I’m going to put it all. My family is really going to hate me now. 🤪 But here we goooooo! Oh! Shout out to my SIL for the double French braids over Easter bc they always leave the best waves! 😍 #reseller #resellercommunity #resellerlife #resellerpallet #growth #amazonseller #amazonfba #amazonfbaseller #fbaseller #isellthingsonline #failfast #entrepreneurdiaries #growth #pallets #inventorypallet #fuckaroundfillapallet (at Cinco Ranch, Texas) https://www.instagram.com/p/CNXl9KDFHMu/?igshid=11dt9m1ay31xt

While there will always be people in front of you and behind You. Sometimes there is Only 1 Opportunity, and Winner Takes All. 𝗢𝘃𝗲𝗿𝘁𝗮𝗸𝗲 𝗼𝗿 𝗕𝗲 𝗢𝘃𝗲𝗿𝘁𝗼𝗼𝗸. That's the Game. You Win or You Learn. 𝗟𝗲𝗮𝗿𝗻 𝗙𝗮𝘀𝘁 + 𝗗𝗼 𝗙𝗮𝘀𝘁 + 𝗙𝗮𝗶𝗹 𝗙𝗮𝘀𝘁 + 𝗜𝗺𝗽𝗿𝗼𝘃𝗲 𝗙𝗮𝘀𝘁 = 𝗦𝘂𝗰𝗰𝗲𝗲𝗱 𝗙𝗮𝘀𝘁 And Money Loves Speed (If You are Interested...) 😃 #TheOnlyWay #TheOnlyWayIsThrough #TheOnlyWayIsUp #Success #Succeed #Learn #LearnFast #Learning #LearnFaster #BeFaster #FailFast #FailFaster #SucceedFaster #TheRace #WinnerTakesAll #OneOpportunity #Overtake #Outwit #Outlast #Outplay #Outsmart #Winning #Losing #YouWinOrYouLearn #TakeAction #TakeActionToday #TakeActionNow #LearnFromYourMistakes #Improve #Speed https://www.instagram.com/p/CaM_9xzpDlu/?utm_medium=tumblr

Ready for Online Strategy Workshop tomorrow with 22 participants (inhouse). Ingredients: - Strong Mac Mini 32GB - Sony alpha cam - Ipevo document cam - Logitech webcam - Logitech presenter - 2x Dell 24 Zoll screens - LG OLED TV - AirPods - Diffuse lighting - Green screen - Zoom - Miro - mmhmm beta tester ... - Passion - Skills - Growth mindset Open for enquiries. #online #workshop #training #coaching #strategy #designingstrategy #strategybydesign #agility #invinciblecompany #businessmodel #businessmodelcanvas #businessmodelinnovation #portfolio #innovation #experiments #failfast #vuca #pivot #leanstartup #sattlerconsulting #businessinnovator (hier: Munich, Germany) https://www.instagram.com/p/CGAYAQepJC4/?igshid=pahwxf65jriu

From a lifetime of giving it my all, the scar tissue is thick from my failures. But I would not be the man I am today or have the success I get to enjoy if it weren’t for the lessons learned from each scuff on the bottom. But with enough tenacity to get back up, you are boundless by what’s possible! #success #failure #failfast #successquote #quoteoftheday #livingthedream #dreamchaser #beyourbestself #ambitionquotes (at Palm Springs, California) https://www.instagram.com/p/CDJm1LlJpif/?igshid=m5k01aei0dxt

Failure is scary. Starting over is tedious. But they both have their benefits. 📉😨📈 Last night, I was working on a project that just wasn’t coming together. I spent hours trying and failing to get the background to work. Once the individual pieces finally started to fall into place, my computer crashed. Even worse – None of the progress that was made had been saved, and there was no recovery file to be seen. 💻🎨💀 It happens. 💩💩💩 Fail fast and learn from your mistakes. 📉💨💡 Anyways, it’s WIP Wednesday, so I guess I’ll take a moment to part the curtain and show off a little of what I was talking about. I don’t think it is finished (in fact, I may have to completely redo the torso), and after a night of sleep, I think I can do better still with the background, but here is a more fully fleshed out version of a critter that was hinted to earlier this month. It’s an endangered reptile native to Guatemala called Cope’s Arboreal Alligator Lizard. Go ahead and Google that. It’s a beautiful lizard, and like a few other critters I’ve drawn in the past, its home is limited to a very small portion of the globe. 🌲🐊🌳 So what do you think? Any animals you think I should draw? What’s your favorite animal? Do you have any thoughts/feelings/stories about failing/starting over? Leave those answers in the comments below, and have a wonderful day! 😸😸😸 . . . #failure #startingover #fail #failfast #failoften #shithappens #bepositive #positivity #saveearly #saveoften #endangered #animal #animals #arboreal #alligator #lizard #reptile #critter #critters #favorite #favoriteanimal #blossoms #flowers #pink #green #wip #wipwednesday #workinprocess #endangeredanimal #endangered

Last Day of the Surviving Business Failure Series. The simple message is You Can Do It!! You can bounce back from Business Failure into a new venture to success and happiness. #bouncepreneurs #bouncebackinbusiness #survivingbusinessfailure #womenentrepreneurs #entrepreneur #entrepreneurs #entrepreneurlife #business #businessplan #fuckupnights #fuckupnights #fuckupnightsthehague #fuckupnightsamsterdam #failfast #womeninbusiness #entrepreneurship #resilience (at Milford Haven) https://www.instagram.com/p/ByUskuNHAzr/?igshid=2v5bifyl34am

Turned a failed wash n go into a cute and carefree look. Never give up! #failfast #keeplearning #naturalhair (at San Francisco, California) https://www.instagram.com/p/CALYGEPJ0Fl/?igshid=gen87dybe3yy