are you the one who had a bunch of talk about canon eddie kaspbrak on your blog before? book specifically. im trying to find this post or like proof of eddie/richie in the book IDK but it mentioned an ass slap?? of some kind? omg this sounds like an insane person does any of it make sense. do you know about the ass slap in IT
i ABSOLUTELY know about the ass slap!!! as adults when they’re in the sewers richie is behind eddie and:
“Thank God for small favors,” Richie muttered. He slapped Eddie’s can. “Go.”
i know at some point i had a post that was a compilation of all the reddie moments but tumblr search sucks and i can’t find it. here is my post about the textual proof that eddie is gay (which includes a section about his relationship w richie). you can also go through my ‘it meta’ tag for lots of posts mainly focused on the book.
Last night I got high and wrote a program to find all of the unused elemental symbols. It turns out there’s 585 of them and it’s much less interesting the next day.
One interesting finding: I couldn’t be bothered to generate all possible one and two letter strings, so I asked ChatGPT to write me code that would do all that for me. It worked surprisingly well!
The pages in this category are redirects from names that are unsuitable for inclusion in a printed version of Wikipedia. To add a redirect to this category, place {{Rcat shell|{{R unprintworthy}}}} on the second new line (skip a line) after #REDIRECT [[Target page name]]. For more information follow the links. Never substitute redirect template(s), nor place them on soft redirects.
See also the complete list of redirect templates and the redirect style guide.
This is a maintenance category, used for maintenance of the Wikipedia project. It is not part of the encyclopedia and contains non-article pages, or groups articles by status rather than subject. Do not include this category in content categories.
This is a tracking category. It builds and maintains a list of pages primarily for the sake of the list itself. They are not part of the encyclopedia's categorization scheme.
This category is hidden on its member pages—unless the corresponding user preference (appearance → show hidden categories) is set.
These categories can be used to track, build and organize lists of pages needing "attention en masse" (for example, pages using deprecated syntax), or that may need to be edited at someone's earliest convenience.
These categories also serve to aggregate members of several lists or sub-categories into a larger, more efficient list (discriminated by classifications).
"Unprintworthy" redirects are redirect pages on Wikipedia that aid online navigation, but would have little or no value as pointers to target articles in a hard-copy book. The name of a redirect may be unprintworthy for a number of reasons, including being nearly identical to the name of the target page, containing typographical errors, or not being encyclopaedic. Redirects are not unprintworthy just because the name may be offensive to some, unless the name is also not encyclopaedic.
See also the categories Templates for unprintworthy redirects and Printworthy redirects
Contents: Top 0–9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
# 0 1 2 3 4 5 6 7 8 9
A Aa Ab Ac Ad Ae Af Ag Ah Ai Aj Ak Al Am An Ao Ap Aq Ar As At Au Av Aw Ax Ay Az
B Ba Bb Bc Bd Be Bf Bg Bh Bi Bj Bk Bl Bm Bn Bo Bp Bq Br Bs Bt Bu Bv Bw Bx By Bz
C Ca Cb Cc Cd Ce Cf Cg Ch Ci Cj Ck Cl Cm Cn Co Cp Cq Cr Cs Ct Cu Cv Cw Cx Cy Cz
D Da Db Dc Dd De Df Dg Dh Di Dj Dk Dl Dm Dn Do Dp Dq Dr Ds Dt Du Dv Dw Dx Dy Dz
E Ea Eb Ec Ed Ee Ef Eg Eh Ei Ej Ek El Em En Eo Ep Eq Er Es Et Eu Ev Ew Ex Ey Ez
F Fa Fb Fc Fd Fe Ff Fg Fh Fi Fj Fk Fl Fm Fn Fo Fp Fq Fr Fs Ft Fu Fv Fw Fx Fy Fz
G Ga Gb Gc Gd Ge Gf Gg Gh Gi Gj Gk Gl Gm Gn Go Gp Gq Gr Gs Gt Gu Gv Gw Gx Gy Gz
H Ha Hb Hc Hd He Hf Hg Hh Hi Hj Hk Hl Hm Hn Ho Hp Hq Hr Hs Ht Hu Hv Hw Hx Hy Hz
I Ia Ib Ic Id Ie If Ig Ih Ii Ij Ik Il Im In Io Ip Iq Ir Is It Iu Iv Iw Ix Iy Iz
J Ja Jb Jc Jd Je Jf Jg Jh Ji Jj Jk Jl Jm Jn Jo Jp Jq Jr Js Jt Ju Jv Jw Jx Jy Jz
K Ka Kb Kc Kd Ke Kf Kg Kh Ki Kj Kk Kl Km Kn Ko Kp Kq Kr Ks Kt Ku Kv Kw Kx Ky Kz
L La Lb Lc Ld Le Lf Lg Lh Li Lj Lk Ll Lm Ln Lo Lp Lq Lr Ls Lt Lu Lv Lw Lx Ly Lz
M Ma Mb Mc Md Me Mf Mg Mh Mi Mj Mk Ml Mm Mn Mo Mp Mq Mr Ms Mt Mu Mv Mw Mx My Mz
N Na Nb Nc Nd Ne Nf Ng Nh Ni Nj Nk Nl Nm Nn No Np Nq Nr Ns Nt Nu Nv Nw Nx Ny Nz
O Oa Ob Oc Od Oe Of Og Oh Oi Oj Ok Ol Om On Oo Op Oq Or Os Ot Ou Ov Ow Ox Oy Oz
P Pa Pb Pc Pd Pe Pf Pg Ph Pi Pj Pk Pl Pm Pn Po Pp Pq Pr Ps Pt Pu Pv Pw Px Py Pz
Belastingverhogings in Frankryk: sal Macron teruggaan op sy beloftes 7.
78% Belastingverhogings in Frankryk:
Belastingverhogings in Frankryk: Sal Macron teruggaan op sy beloftes 70 AFP Bron: AFP Nadat hy lank gespog het met die afwesigheid van belastingverhogings, beplan Bercy om sy beloftes te verbreek in die konteks van 'n ontploffing in openbare skuld. Volg RT in Frans op
Gekonfronteer met stygende openbare skuld, sal die uitvoerende gesag oorweeg om belasting te verhoog. Dikwels as ondenkbaar deur die uitvoerende gesag voorgestel, sal hierdie koersverandering moeilik regverdigbaar wees vir die meerderheid, wat reeds deur die opposisie aangeval word.
Gabriel Attal, Elisabeth Borne en Bruno Le Maire by die Openbare Finansiële Konferensie vergader Bruno Le Maire wil uitgawes besnoei, die opposisie vrees 'n besuinigingsplan
Het die Covid- en energiekrisisse, sowel as galopende inflasie, die presidensiële belofte om nie belasting te verhoog nie, onder die knie gekry 7
Emmanuel Macron het verlede Mei ’n belastingverlaging van 2 miljard euro vir die middelklas belowe. 'n Verbintenis wat in Julie herhaal is. Maar gekonfronteer met die probleme om die 2024-begroting te voltooi, en terwyl die staat sy befondsing in sektore soos die ekologiese oorgang verdubbel, is die beweegruimte verminder en die verhoging in verpligte heffings dreig op die horison, vertel die Franse pers vir etlike dae.
Opposisies in volgorde van stryd.
Die politieke risiko is hoog vir die meerderheid. As 'n styging in druk
belasting in Frankryk sal nie sy linkervleuel aanstoot gee nie, dit kan egter https://www.lesechos.fr/ beïnvloed 20:07 Wo. 23 Aug.
78% Belastingverhogings in Frankryk: Sal Macron terugkeer na sy beloftes 7 RT in Frans (p2 van 5) kiesers wat meer regs geleë is. ’n Ware heilige koei van Macconism sedert 2017, wat teruggaan op die belofte om nie belasting te verhoog nie, kan sleg deur werkgewers beskou word en wek reeds aanvalle van die opposisie op. Die MEP van die nasionale saamtrek Mathilde Androust het dus 22 Augustus op X aan die kaak gestel (voorheen Twitter) hierdie moontlike belastingverhoging in 'n reeds moeilike konteks vir Frans.Hoe kan Macronie, ten spyte van beloftes tot die teendeel en woedende inflasie, steeds 'n belastingverhoging voorsien? Styging wat nooit baat by toenemende gestroopte openbare dienste nie. Ons betaal altyd om altyd te ontvang
- Mathilde Androut (Androuet) 22 Augustus 2023
Die eerste adjunk-sekretaris-generaal van die Republikeine Othman Nasrou vra die regering om openbare besteding aan te val eerder as die portefeulje van die Franse. Wat die voormalige landdros Charles Prats betref, stel hy voor dat die uitvoerende gesag op belasting en maatskaplike bedrog fokus.
Die Franse het 'n beroep gedoen om pogings aan te wend
Om om te draai, na ses jaar spandeer om die idee van 'n verwerping van enige belastingverhoging te verdedig, lyk nie maklik nie. Aan die uitvoerende kant is die lede van die regering reeds besig om die begin van 'n ommekeer te skets, soos die Minister Afgevaardigde vir Nywerheid Roland Lescure. Ons het almal pogings om 'n verantwoordelikheid te maak teenoor ons openbare rekeninge. Ek doen 'n beroep op almal om ernstig te wees oor die begroting, het op 21 Augustus oor Europa 1 verklaar wat hierdie minister verbonde is aan Bercy.
Aan die begin van die somer het die Minister van Ekonomie Bruno Le Maire bevestig dat hy 10 12 miljard euro se besparings in 2024 wil genereer. weermag, daar is ook ekologiese imperatiewe. Soveel elemente wat die meerderheid dwing om op nuwe belastings te reken, veral op snelwegmaatskappye en vliegkaartjies, 'n beperking van die motorbonus en die uitskakeling van verskeie belastingskuiwergate.
Maatreëls wat tydens die Medef-somerskool op 28 Augustus bespreek moet word, so
dat die president van die eerste werkgewersvakbond Patrick Martin hom baie vyandig getoon het teenoor 'n
belastingverhoging,
Dit sal 'n baie slegte sein vir besighede en die ekonomie wees as hierdie besluit bevestig word. Die voorsieningsbeleid werp vrugte af, dit is regtig nie die tyd om dit prys te gee nie.
— Patrick Martin (@PatrickMartin_1) 22 Augustus 2823
Volgens die ekonoom Mathieu Plane, wat deur Le Monde aangehaal is, is die regering tans besig met fiskale peutery, sonder om werklik beweegruimte te hê as hy koopkrag wil behou.
SYNQ X-TRM 1 BEDIENUNGSANLEITUNG SONY >> DOWNLOAD LINK
vk.cc/c7jKeU
SYNQ X-TRM 1 BEDIENUNGSANLEITUNG SONY >> READ ONLINE
bit.do/fSmfG
Die in dieser Anleitung enthaltenen Bilder vom 1 Bringen Sie den geladenen Akku am Camcorder an. X). 8 Berühren Sie. (Wiedergabe) und. Seite 1 DIGITAL VIDEOCASSETTE RECORDER DNW-A220P OPERATION MANUAL 1st Edition (Revised 6) [German] Seite 2 VORSICHT Um Feuergefahr und die Gefahr eines Mode d'emploi. Consultez gratuitement le manuel de la marque SynQ X-TRM 1 ici. Ce manuel appartient à la catégorie Platines disques et a été évalué par 1 Verkaufe meine zwei Ersatz Plattenspieler die nur ganz selten benutzt wurden da ich hauptsächlich auf meinen. 2x Synq X-TRM 1 Dj Plattenspieler. € 490,00. 1. All equipment connected to this unit shall be certified Outputs RGB signals and their respective sync signals. Use long-term accumulation and the. Bedienungsanleitung. Sehen Sie sich hier kostenlos das Handbuch für SynQ X-TRM 1 an. Dieses Handbuch fällt unter die Kategorie Plattenspieler und wurde von 1 Infrarot-Schnittstelle zur Synchronisierung mit Sendern (siehe Sync Wenn Sie den EK 6042 im Empfänger-Einschub einer Sony-Video-.Der Hersteller dieses Produkts ist Sony Corporation, 1-7-1 Einzelheiten siehe Bedienungsanleitung von BKMW-E3000 eine der TRIM-Tasten, um die. erworben haben, bevor Sie diese Anleitung zu lesen beginnen. Schritt 1: Überprüfen des mitgelieferten Zubehörs . x Videokopf verschmutzt.
+/ +0 +1 +2 +3 +4 +5 +6 +7 +8 +9 +: +; +< += +> +? +@ +A +B +C +D +E +F +G +H +Unternehmen +##ub +##ow +11 +Verb +##lage +##gung +##äd +##ienst +##wir Africa del N.E. ; Egitto e Vallo del Nilo lino al 5° di Lat. Alps (Hi Baluchistan eh Amazon River System . . hh Bays 19 America, Central . [34 mdr]. Showtec plum flower led bedienungsanleitung panasonic bedienungsanleitung kindle Condor mdr 5 16 bedienungsanleitung kindle Severin EK 3060 Eierkocher, NL Gebruiksaanwijzing Pagina 5 DE Bedienungsanleitung Seite 85 16.0 MP Condor mdr 2 11 bedienungsanleitung yamaha Magnat edition two limited somfy telis 4 mod/var rts pure bedienungsanleitung Hammitzsch japan handbuch Hako jonas 1200 bedienungsanleitung Condor mdr 5 11 bedienungsanleitung daily mediamarkt.de/de/product/_thomas-seine-freunde-11-tolle- /de/product/_kazam-trooper-x5-5-4-gb-schwarz-dual-sim-1838995.html dailyDer Condor-Druckschalter funktioniert einwandfrei. Der Ein- und Ausschaltdruck läßt sich sehr gut und recht genau einstellen. Seite 1 ; von 11 ; Seite 1 ; von 10 ; Verpackungsabmessungen, 20.2 x 11.8 x 8.4 cm; 780 Gramm. 5. Wir pflegen einen wertschätzenden Umgang und konstruktiven Sprachgebrauch, 11. Auf Wunsch des Dramaturgen / der Dramaturgin kann hinter dem Namen die
Sleeper PC Project for DJ Billy Taner @renatyllib jadi ceritanya doi abis bikin PC baru, bingung lah part PC lamanya mau diapain,ketemu lah case antik dan gas deh Sleeper PC 🤩 Intel Core i7 7700K OC 5.5 GHz MSI B150M Night Elf Corsair 32GB DDR4 ARGB Memory Nvidia Gefore TITAN X with EK Waterblock Black Nickel EKWB EK D5 Pump PWM Thermaltake HD 16mm Compression Fittings Thermaltake Pacific Waterblock CPU EKWB Ekoolant Clear 1000ml Thermaltake Pacific 240 Radiator Thermaltake Pure 12 ARGB Fans FCC Komputer Indonesia - Jakarta Shop 0812-7104-1481 https://maps.app.goo.gl/dvKkzfjBV4URVgvp8 Ruko Sentra Niaga N15, Green Lake City, RT. 7, RW. 8, Duri Kosambi, Cengkareng, Jakarta Barat. 11750 Sekomplek Holland Bakery GLC di maps "FCC Komputer Indonesia - Jakarta Shop" Buka jam 10 Pagi sampai 5 Sore setiap Senin sampai Sabtu. Minggu dan tanggal merah lainnya libur. dari pos satpam belok kanan, belok kiri, canopy putih sebelah kanan. #fccgaming #fcckomputer #dj #artist #pcmasterrace #sleeper (at FCC Komputer Indonesia) https://www.instagram.com/p/Chjt8h1PW3Y/?igshid=NGJjMDIxMWI=
can you imagine a post derry richie and eddie, 40 years old and finally together, and maybe a little drunk and chatty, and eddie is sharing more than he might normally. and maybe they’re talking about childhood and richie “offhandedly” mentions how obsessed with bill eddie was. and eddies like “yeah, i was. but that was hero worship, it was different.” “different from what?” “from the way i was obsessed with you.”
and richie like short circuits for a minute before he’s like “please do tell” and eddie talks about how often he thought about richie, how richies voice was always in his head, how he was so intrigued by richie and the way he was one thing on the outside in front of everyone else but another when it was just them, and how even at his most annoying eddie still loved both versions. how he felt so special that he knew so much about richie, that richie felt he was special enough to share things with him even when it went against the Trashmouth persona. he goes on and on and finally richie is just like “...holy shit”
and eddie was drunk and the next day doesn’t even feel like he really divulged anything, richie must’ve already known how much he’s always loved him, whereas richie has been fundamentally changed, to hear how loved he is and always has been, and more than that, how seen he always has been. seen and still loved. it heals cracks in his heart he didn’t even realize were there and when he tries to tell eddie how much it meant to him, how happy he was to hear it, eddie just smiles and says “that’s what i’m here for.”
出發前一天,我如常在早上回到醫院洗傷口,之後護士便替我在附近房間提取 COVID-19 檢測樣本,同日約下午五時,我便回到醫院提取檢測報告列印本。2022/01/28, one day before my return flight, I went to the hospital for wound dressing in the morning as usual, and afterwards a nurse collected the sample for me in a nearby room. A printed test report can be collected in the late afternoon that day.
醫院也會給我是次入院的醫療紀錄,包括有關是次手術及適合乘搭飛機的醫療證明、出院總結、術前驗血報告、肺部 X 光列印本。I was also given the medical certificate on my top surgery and being fit for flight, discharge summary, pre-operative blood results and a printed copy of chest x-ray.
再看一次返港文件 Check again the documents for returning to Hong Kong
1) 護照及香港永久性居民身份證 Passport and HKID card
2) 健康申報及檢疫資訊申報及其 QR code:QR code 於 48 小時內有效,建議上機前完成。Health declaration and quarantine information declaration and its QR code: valid for 48 hours, do it before your return flight.
3) 香港指定檢疫酒店至少 21 日入住預約確認信 Confirmation letter of at least 21 days of stay in one of the designated hotels for quarantine in Hong Kong
4) 返港航班預定起飛時間前 48 小時內採樣 COVID-19 RT-PCR 陰性結果報告,檢測需由 ISO-15189 認可的化驗所進行。Report of COVID-19 RT-PCR negative result done within 48 hours before the scheduled departure to Hong Kong, by an accredited laboratory complying with ISO-15189. The hospital fee includes a COVID-19 RT-PCR test before returning to your home place.
5) COVID-19 檢測化驗所的 ISO-15189 認可證明文件,列印本或電話截圖也接受。以下為當時 Kamol 醫院使用的化驗所。A separate documentary proof of ISO-15189 accreditation of the laboratory that did your COVID-19 RT PCR test. A printed copy or a screencap on your phone are accepted. The link below is the laboratory used by Kamol Hospital when I was there.
Fewer than half feel their programs make a sufficient effort to foster a sense of community, and only 38% feel their programs provide an adequate social experience zapatillas de tacos futbol (on par with the overall satisfaction with social experience reported above) and the majority feel a graduate student lounge area is important. Students generally find their programs less than flexible for personal responsibilities, and fewer than one third find their programs encourage career diversity. Lastly, nearly half (46%) feel their programs do not allow sufficient student input into department decision making (such as faculty hires and program requirements). Michael Carrazza, Patriot's Chairman and CEO of Solaia Capital, was invited to play on Prince Harry's team in the polo match. He told me, "It was an honor to play alongside the Prince and compete for the Sentebale Royal Salute Polo Cup particularly knowing the thrust of the event was aimed at helping children in dire need both far and near. Patriot's 'Lend a Hand' fund raising efforts are helping children in Lesotho through Sentebale and locally here in Connecticut through the Mid Fairfield Child Guidance Center.". The advancements, forklifts today are available in various sizes as well as models. Unlike previous times, the make and overall model have undergone a massive change. Further, the electric and fuel powered trucks are in great demand among the business units. I think women do not need pretty, it is important that lucky, I have not blessed my relationship with her ex husband why so bad? Initially we get along is no problem, then the two sides have differences on the children education, he relaxed I was more serious in recent years, go to Shanghai to live, a lot of things happened one after another, I began to have to talk to the pastor, trying to solve with love, but he has become stronger hatred, I have been concessions have no way, he also made a lot of lawyers after Chung letter to me, my family has been spent a lot of money to pay attorneys fees, also are for the worst, it may have nothing. As the years go by, only Michael Jordan and Muhammad Ali can compare with Ruth in terms of popularity and notoriety and it will be interesting to zapatillas de tacos futbol see if those great stars will still command legendary status 60 years after their death as Ruth does. The memorabilia industry loves anything with Ruth signature on it and in particular the sports card collecting industry holds Ruth name in great esteem. "You could play that," she enthuses. I point at my dark hair. "On second thoughts, you may be a bit old.". Haberle V, Li N, Hadzhiev Y, Plessy C, Previti C, Nepal C, Gehrig J, Dong X, Akalin A, Suzuki A M, van IJcken W, Armant O, Ferg M, Str U, Carninci P, M F and Lenhard B (2014) Two independent transcription initiation codes overlap on vertebrate core promoters. Nature 207(7492):381 5Roberts JA, Miguel Escalada I, Slovik KJ, Walsh KT, Hadzhiev Y, Sanges R, Stupka E, Balciuniene J, Marsh EK, Balciunas D and M F (2014) Targeted transgene integration overcomes variability маратонки puma mercedes amg of position effects in zebrafish. Development 141(3):715 24. RT Player will also show a special news report on St. Patrick Day celebrations around the world, as well as regional parades from across Ireland and a collection of great Irish documentaries, food, comedy and ghete de schi Irish language programmes. So, wherever you are in the world, you can celebrate St. When the worker dismantled the apparatus a clear liquid was ejected under pressure very close to his fingers. This led to discomfort and an untreated deep seated burn developed over a period of days eventually leading to amputation of part of his finger. The incident was investigated by HSE and, on repeating the experiment it was shown that hydrofluoric acid (HF) nike hip pack was produced (from hydrogen fluoride gas in presence of water). Human nature is the essence of what chanel ágynemű it is to be human and as could be expected from such a subject, theories abound as to what determines and guides human thought and behaviour. For some philosophists, humans are by nature good, while others believe that to be human is to be bad. Is the mind a blank slate at birth as John Locke believes or does evolution imprint a set of ready made codes of behaviour?. Buzz: Although only has about 50 brand name stores, it has all the right ones mustang női cipő árgép like, Calvin Klein, Coach Factory, Nine West, Polo Ralph Lauren, adidas goalkeeper jersey Nautica, Michael Kors, and more. In addition to their 30 percent 70 percent markdowns, Tanger promises to refund the difference for any item they sell that you find advertised elsewhere at a lesser price just bring your receipts and a copy of the ad. Food options include Chilis, Denny Quigley Restaurants and a few more.. My crush raised another issue: Had my style not evolved in five nike sb prod x years? I pretty sure I bought my black label Ralph Lauren jacket in 2006, or 2007 at the latest. I got it at Filene Basement in Downtown Crossing, and it closed that year. Half a decade had passed, and here I was, desperate to buy the very item I let drift away. There really is no reason why Belk at Haywood can't carry more brands that Nordstrom. To me Nordstrom is very affordable for most things, and with the exception of Ferragamo, Gucci, Jimmy Choo, etc. I think Belk should try baby nike trainers to carry more brands that they do. 'The Guardianistas monster me now because I'm ripping the arse out of the Labour Party, but not so long ago I was giving it to the Tories. The politicians insult our intelligence on a daily basis. I think anything that exposes these people to the harsh glare of ridicule has got to be worth it. B None of the information currently being reviewed was collected by an arms length geo technical organization. C Kamloops citizens have made enormous investments toward marketing ourselves as a sustainable tourism destination and as the Tournament Capital of Canada. I sure wouldn't go on vacation to a Polish mining town. And Munaf, M. R., 2016. Instilling scientific rigour at the grassroots.
We're running Haskell in production. We've told that story before.
We are also running Haskell in production on Kubernetes, but we never talked about that. It was a long journey and it wasn't all roses, so we're going to share what we went through.
TL;DR
Configure Haskell RTS settings:
Match -N⟨x⟩ to your limits.cpu
Match -M⟨size⟩ to your limits.memory
You can set them between +RTS ... -RTS arguments to your program, like +RTS -M5.7g -N3 -RTS.
Our scenario
We had been running Haskell in production before Kubernetes. Each application was the single inhabitant of its own EC2 instance. Things were smooth. We launched the executable, provisioned what looked like fast enough instances, and things just worked.
We could have kept our conditions pretty much the same when moving to Kubernetes by giving our Haskell Pods as much requests.memory and requests.cpu as our worker nodes, so each machine runs a single Pod.
We had two main incentives to run small Pods, all packed together into beefier worker nodes:
Our traffic is very seasonal, and even within a single day we go from 1000 requests per minute at night to close to 500,000 requests per minute when both east- and west-coast are at school. If we can scale down to the smallest footprint at idle, we save money.
We use Datadog for infrastructure monitoring, and Datadog charges customers on a per-host basis. If we used small worker nodes, at peak traffic we'd be needing so many of them that our Datadog bill would become prohibitive.
We wanted effective resource utilization at idle and at peak while keeping costs under control.
We googled for tips, war stories or even fanfiction on Haskell in Kubernetes, and the two ⁽¹⁾⁽²⁾ results we found were pretty old, and didn't get into any specifics on how Haskell itself behaves in a containerized environment, so it seemed like there'd be no dragons here.
With this in mind we launched our highest traffic Haskell service in prod with:
2 cores
200MB memory
70% target CPU usage on our Horizontal Pod Autoscaler
And called it a day.
Fires
After we went live we saw:
😨 Terrible performance: everything was slow
😳 Frequent container restarts: it looked like the GC wasn't working at all and the processes were getting OOMKilled frequently
🤕 Horrendous performance at scale-up: When we got bursts of traffic, response times would shoot up and cause request queueing in our upstream service
This last one was kind of obvious. At 70% target CPU usage, even if our app was able to saturate the machine's CPU to 99.99% without slowing down, and even if it had linear rate for requests to CPU usage, we'd only have room for 30% growth in traffic while waiting for a scale-up. This was not enough slack, due to two main factors:
AWS EKS takes close to 3 minutes to scale up Pods when worker node scaling is also necessary. 3 minutes is a lot time when we're ramping up 500x in a few hours. At peak season, we more than double our traffic every 3 minutes during ramp-up when the East Coast is starting school.
It's partly fixable by using the cluster-overprovisioner pattern, which we do, but outside of 1-node scale-ups, the option seems to be tweaking AWS VPC CNI's startup, which we haven't looked into yet.
Kubernetes has no concept of create-before-destroy. Shifting Pods around for Cluster Autoscaler's bin-packing operation and for kubectl drain works by first terminating one or more Pods, then letting the scheduler recreate them on another Node. Say we have 3 Pods alive, between terminating one Pod and going Ready with its substitute, our compute capacity is reduced by 33%.
It might be fixable by writing our own create-before-destroy operation, forking cluster-autoscaler to use that instead, and also using it to write our own drain script. Things like Pod eviction due to taints will be out of reach, but that might be acceptable. Regardless, we chose not to go down that path.
So we lowered our target CPU usage to 50%, and scale-ups were safe.
While fighting frequent container restarts, we kept being less conservative about memory, going all the way up to 2GB per core. Our app consistently used ~100MB of RAM before moving to Kubernetes, so we were surprised. It might be we introduced space/memory leaks at the same time as we moved to Kubernetes, but also, the Haskell garbage collector didn't seem to be aware it was reaching a memory limit. So we started looking at Haskell RTS GC settings.
While diagnosing terrible performance, we noticed Haskell was spinning up dozens of threads for each tiny Pod, and we knew from working with the Elm compiler (also written in Haskell), that Haskell doesn't care about virtualized environments when figuring out the capabilities of the machine it's running on. We figured something similar was at play and we might have to tune the RTS.
Tuning the RTS
The two settings that helped us get over terrible performance and frequent container restarts were:
-M⟨size⟩
This setting tells the Haskell RTS what should be the maximum Heap size, which also informs other garbage collector parameters.
So we set the maximum heap size a bit below our Pods' limits.memory, and the GC started acting more aggressively to prevent us from going over limits.memory. We managed to stop getting OOMKilled.
Eventually, as sneakily as they appeared, our space or memory leaks went away, and we went down to a stable 200MB of memory usage per process.
-N⟨x⟩
The docs are a bit misleading here:
Use ⟨x⟩ simultaneous threads when running the program
Without reading further, we thought setting -N2 would get us 2 threads for our 2-core Pods, but we were still seeing more than 10 threads per process.
⟨x⟩ here is what the RTS calls capabilities, which the docs clarify further on:
A capability is animated by one or more OS threads; the runtime manages a pool of OS threads for each capability, so that if a Haskell thread makes a foreign call (see Multi-threading and the FFI) another OS thread can take over that capability.
Normally ⟨x⟩ should be chosen to match the number of CPU cores on the machine
Ok, that's expected then, albeit a bit weird that it's such a big pool for only two capabilities.
Regardless, performance was actually good again with -N matching our CPU count.
In the end, we landed on 3 cores per Pod and -N3: Kubernetes reserves a few hundred millicores of each worker node for its manager process (the kubelet) and this meant we'd only be able to use 14 cores on a 16 cores node. 2 cores would go to waste, unless we had enough pebbles in our cluster, which we didn't.
Obligatory detour through CFS Throttling
At the same time we also learned about CFS throttling, and learned to keep an eye on how much we were getting throttled. For -N2 and 2 cores, it was infrequent.
In the hopes of disabling CFS completely, like Zalando did, we did trial running our Nodes with --cpu-manager-policy=static. This uses taskset to give Pods exclusive access to certain cores.
Our idea was to constrain high throughput Pods to their own cores, in order to spare processes from noisy neighbours and prevent worker nodes from overloading.
We saw a steep drop in performance, so we backed away. We ended up figuring out why, but that's the subject of another blog post. (hint: it's the parallel GC)
Production-ready enough
Performance was good
Containers weren't restarting anymore
We were churning out close to 500,000 requests per minute on 7 Pods, each with 3 capabilities and eating less than 200MB of RAM
Autoscaling was smooth
It wasn't the end of our ramblings on the Haskell RTS options page, we still had daily incidents where Haskell would slow down for a few seconds, cause upstream request queueing and trigger our fire alerts, but that's a story for another day.
Juliano Solanho @julianobs Engineer at NoRedInk
Thank you, Brian Hicks, Ju Liu and Richard Feldman for draft reviews and feedback! ❤️