Turn / LTY

vampire!au Taeyong x Y/N 6k vampire thingy where weird stuff just seems to happen, but you don’t care as long as you were close to Taeyong // no warnings

Honestly, you don’t know a lot about your town’s history. Still, you did know that vampires and humans lived in harmony together. You remember something about a truce many moons ago where vampires vowed not to bite a human in exchange for a peaceful life. Even though you have never really heard of anyone being randomly bitten for the past fifty years, humans were still wary of the vampires. 

One side of your town was bright and filled with people living their carefree lives, and the other was more dark and forbidden territory to humans. Vampires actually didn’t care if you would go over to their side. However, the elderly people in your town still remember the times when simply walking through their neighbourhood meant being bitten, or worse being turned into one of them.

There was only one case five years ago where a female vampire had turned your neighbour into a vampire, but this was only because they were in love. You always thought it was kind of strange, as she was probably over a hundred years old and your neighbour was only twenty at the time. What could they have possibly talked about? You still remember everyone being upset by the news, and local stores closed for a whole week because everyone thought the vampires went loose again. They didn’t, of course. After a week or so, everything had gone back to normal, and people seemed to have forgotten about your neighbour. As if he had never existed in the first place. 

You loved walking through the vampire part of town, as it was more deserted and gave you some time to think. You also needed to go through the quiet streets to get to your classes. You never really liked taking the bus as it was too crowded, and you definitely didn’t want to take the long route. Your only option was to go through the peaceful streets filled with weird antique shops and high-tech stores. You had entered one of their shops once, and it was as close to normal as could be. You couldn’t tell if the lady behind the register was a vampire, but in the end, it didn’t matter. They were simply living their life in the same peaceful manner as a human would do. You didn’t understand the fuss about them being mean and secretive. Vampires used to be human too, and the vampires in your town wanted to live an ordinary life. 

You didn’t know many vampires. You only knew of the lady in the shop who could or could possibly not be a vampire. But there was one person you suspected of being a real vampire, and his name was Taeyong. He lived a bit outside town, like you, where there was no strict vampire or human territory. He had moved a little over a year ago to go to your local college together with some friends. He was the brooding and mysterious character who had fascinated you since day one.

At first, you didn’t consider him being a vampire, but there were some giveaways if you paid close attention. He was pale, and this might not a big surprise as everyone in your town was remotely pale due to the lack of sun your city saw, but he was exceptionally pale. As if he was sick, but other days he had a healthy glow and noticeably more energy. He was also too smart to go to your small local college, and this you knew because you shared a lot of the same classes. You only talked to him once, and it wasn’t a success, to say the least.

It was one of your first day of college, and you had never seen Taeyong or any of his friends. Classes were over for the day, and you took your usual route through scary vampire territory. You had bumped into each other, and it was not the romantic kind of walking into each other. He had been running, his eyes crazy, and it scared you. It was never a good sign when someone was running through this part of town.

“Watch where you’re going.” He hissed at you, making you feel small and insecure. As if you were an animal stuck in his trap.

“I’m sorry.” Was the only thing you could say while you were avoiding his intense stare. You had always heard people talk about a vampire’s stare being mesmerizing. Still, you never realized how consuming their eyes were. It seemed to be the only thing you could focus on. It was hard to not look into his eyes, but you fought the urge.

“You should go home. Don’t talk to strangers.” He finally ordered you with concern laced in his tone. You looked up with confusion in your eyes, but that’s where you made a mistake. His eyes were dark brown, almost black-like, and it somehow didn’t scare you.

“My mom told me I shouldn’t talk to strangers, and you’re definitely strange.” You told him, and you cringed at your awkward statement. Were you trying to prove to him that you were not scared because you were, but not of him. He simply laughed at your comment before running off to wherever he was going, leaving you alone and confused. That’s when the small obsession with Taeyong began. You wanted to figure out if he really was a vampire or not, because how else you could explain your speeding heart and moist palms whenever he was near?

You had worked on a project with some of his friends, and you  still couldn’t tell if they were vampires or not. His friends ate the healthy amount of food a human boy in their twenty’s would, and they were actually approachable. You could talk to them, and you could find them at parties chatting someone’s ear off. You actually befriended one of his friends, Mark. He was the kind of person everyone got along with, including you. Taeyong was also present at these parties, but he was always standing somewhere alone in the corner. He preferred to be alone, but girls periodically tried to make a move on him, and he never seemed pleased by their presence. It was quite a hilarious scene to uphold. How the girls attempted to show their cleavage while acting tipsy to get his attention. He never looked at them, disgust expressed in his orbs while he sipped from his drink. 

You and Mark had gotten pretty close since your school project. He would take you out for lunch or would invite you to hang with some of his other friends. It was clear to anyone that you and Mark had this special bond, and both your friend groups accepted the other. He even threw you a little surprise birthday party at your favourite coffee shop. He had invited some of his friends and your friends, and it was one of the most enjoyable afternoon’s you had ever had. 

Mark had given you a birthday card where he mentioned he would always protect you, and he would even hide a body for you if necessary. He was a good friend, and you were glad he was in your life. You never knew a day would come where you would actually take him up on his offer. 

“Hi Y/N!” Mark immediately said when he picked up his phone. He seemed happy and carefree, and you hated to be the person to ruin his mood. Every time something remotely unfortunate happened to you, Mark would try everything in his power to make you feel better. You kind of felt regretful about depending on him so much. You wished you could be the same to Mark, but he never gave you a chance to worry about him.

“Mark, I…” you started, but you were in a state of shock and immediately started crying. Mark noticed your distressed behaviour, and you could hear him saying something in the background. 

“Where are you? What happened? Y/N?” Mark asked you, and you could hear him getting ready to come and pick you up. He was the only person you could depend on. You trusted him 100 per cent. 

“Second Street, corner shop.” Was the only thing you could say through your tears, but Mark knew enough. You heard a door slam, some keys jingling and a motor being being brought to life.

“Give me five minutes.” He said before hanging up, and you quietly waited for him to come and get you. You didn’t want to hang up, but you also weren’t in a state to have small talk. You simply stood there with your bike in hand. 

“Y/N!” Mark yelled through the open window of his car, but he wasn’t driving. You couldn’t see who was driving the vehicle, but you were glad he had finally arrived. It felt like you had waited hours for him to get here, but in reality, it had taken him only three minutes. 

“I think I killed it.” You whispered monotonously, and Mark looked worried by your lack of emotions. You pointed toward the small cat that was still stuck under the wheel of your bike. It didn’t really look alive, and you couldn’t remove your bike out of shock. You had stood frozen in your spot for the last half hour, chances were getting slimmer for the cat to be alive.

“You know, I was joking on that birthday card when I said I’d help you hide a body.” Mark tried to lighten the mood, but you simply presented him with your deadly glare. He seemed relieved by your typical behaviour before taking your hands off the bike to remove the murder machine.

“What happened?” A voice behind you asked, and you immediately turned around in shock. It was Taeyong, and he had probably driven Mark to the murder scene. It was the first time you had heard him speak in close proximity, and it gave you chills all over your body. This was not the time to be aroused by someone’s voice. You were currently the lead suspect of a murder case.

“She hit a cat with her bike.” Mark sighed, and relief flashed over Taeyong’s face. Why did he look so relieved? Had he been worrying about you too? 

“Are you alright?” Taeyong asked you while he moved closer to you. Mark was now examining the cat, and you were glad Taeyong was distracting you from the bloodless scene. His eyes were deep brown and merely mesmerizing, you almost forgot about the situation. Almost.

“No.” You simply said, making Taeyong chuckle lowly. You never really liked to play the damsel in distress, but you couldn’t hide your feelings. You needed their help, and you didn’t care how pathetic it made you look. You had never even killed a fly, and now you had accidentally murdered a cat.

“Do you want to bury the body?” He asked you, and all you could do was nod your head. Mark looked questionably at Taeyong but seemed to make peace with the situation. He simply shrugged his shoulders, trying to think of a solution.

“Okay,” Taeyong said before he went back to his car to grab a towel out of his gym bag. Mark stood by your side to shield you from the scene while Taeyong wrapped the lifeless body in his blue towel. 

“There isn’t a place to bury it here, though.” Mark thought out loud to Taeyong. They were looking at each other as if they were speaking through their minds. 

“We can do it in our garden, I suppose.” Taeyong finally stated. The boys turned toward you expectantly. 

“Do you want to be there?” Mark asked you softly before wrapping his arm over your shoulder. You shuddered at the contact, realizing how cold it had gotten outside. Taeyong had seemed to decipher your reaction and quickly grabbed an extra sweater out of his bag. 

“Yes.” You finally said when Taeyong had given you his sweater. You gladly accepted and put on the extra layer of warmth before following them toward the black car. The piece of clothing smelled freshly washed, and you couldn’t help but relax in the backseat while a dead animal was in the trunk. How did you always get yourself in these weird situations?

You had never gone to Mark’s house, so you did feel a bit awkward because of the situation. The only thing you knew was that he had three housemates, one of them being Taeyong. The others you had briefly met, and you do remember one of them being at your surprise birthday party. You think their names were Jungwoo and Doyoung. 

The moment the car parked in its rightful place, two guys stormed outside with worried expressions. When Taeyong opened your door, and when you made your way out, everyone seemed relieved. Again, why was everyone so concerned about you? You barely knew these people.

“What happened?”

“How’s Y/N?”

“Everything alright?”

Were a few of the questions they immediately fired toward the three of you. Taeyong looked at Mark as if to say he should handle the nosy boys, while he opened the trunk of his car. He carefully grabbed the cat wrapped up in a towel before walking out to their backyard. Everyone silently followed him, not knowing what was in the towel or what the plan was. 

Taeyong instructed his roommates to dig a hole at the edge of the garden while Mark kept you company. No one questioned Taeyong but simply followed his order. You were mesmerized by his powerful vibe, and you couldn’t help but follow his every action. How he carefully placed the dead cat on the grass to adjust the towel. How he sometimes secretly glanced toward you and Mark to check if you were still doing okay. 

When the other two boys were done digging a hole in the ground, they awkwardly awaited further instructions. Taeyong simply looked at you to see if you were ready. He probably expected a simple nod so he could place the cat in the hole, but you wanted to say a few words.

“We are gathered here today to mourn the loss of a suicidal cat.” You started, and you could hear Mark hold back his laughter, his body vibrating next to you. The other two boys simply stared at you. Taeyong had to suppress a smile, and you wished you had seen him smile earlier. It was a beautiful sight to behold.

“I don’t know if he or she is a loner, or if they belong to anyone. I’m extremely remorseful of my actions, and I hope that if there is a God that he or she can forgive me. It was unintentional. May you rest in peace.” You finished your speech. Taeyong noticed and carefully placed the cat in its rightful place in their garden. The other boys skillfully closed up the hole while you softly let tears fall from your eyes. Mark noticed and gave you a tissue he kept in his jacket. He had gotten used to you crying over silly little things, so he always carried an extra tissue just in case. 

You gladly accepted the tissue before wiping your eyes. You realized you had put on make-up this morning, and you probably looked like a raccoon. You looked at Mark, and he simply nodded his head as if he just read your thoughts. 

“Shit.” You muttered under your breath, making Mark laugh. Blame it on the aftermath of shock, but you started laughing along with him. Taeyong and the other roommates looked at you with hundredth questions in their eyes, but you and Mark simply kept laughing like mad people. 

“Is she okay?” Someone asked Taeyong softly, but he simply smiled at you when he thought you weren’t looking. 

“We should get her inside before anyone sees her.” The one named Jungwoo suddenly said firmly. Taeyong simply nodded his head before he took your arm and rushed you inside. You didn’t understand what the issue was of you being seen in their garden, but you didn’t question them. They had helped you without asking you, so you returned the favour. 

“You can stay here tonight. You can sleep in my bed while I sleep on the couch.” Mark told you while he grabbed some water for you to drink. Jungwoo was ordering some pizza while Doyoung and Taeyong were softly talking in the living room. It was weird to hang out in their house when they clearly never invited anyone before. 

“Where is the bathroom?” You asked Mark after drinking the glass of water he had given you. Mark pointed toward the direction of the bathroom, and you followed his instructions. 

You were surprised by your appearance in the mirror, and not in a good way. Your face was swollen from crying, and your skin was pale from the shock. Your mascara was everywhere except on your eyelashes. To summarize, you looked like a walking mess, and you finally realized why everyone kept asking if you were okay. You undoubtedly didn’t look okay. You wiped the remaining make-up off your face and splashed your face with water, hoping to make yourself look more alive. When you were done, you made your way back toward the kitchen, but an argument was going on in the kitchen. It was about you.

“Why did you bring her here? It’s dangerous for humans to be seen around here.” Jungwoo stated firmly, and you could feel the tension rising. 

“She was alone and helpless. We couldn’t just leave her.” Mark defended himself. You hated the fact you had put him in trouble again. He was always helping you while you kept putting him in awkward situations.

“You could have driven her home.” Doyoung piped in.

“I expected this behaviour of Mark, but I didn’t think Taeyong would be so stupid,” Jungwoo said irritated. Him dropping Taeyong’s name made your heart race, and you couldn’t understand why your heart was acting this way. 

“I think she’s back,” Taeyong stated, and you had a weird feeling that he knew you were eavesdropping as if he sensed you being near. 

“Y/N I hope you like Hawaiian pizza,” Jungwoo said when you entered the kitchen, acting as if you didn’t hear the strange conversation. You simply nodded your head when you stood next to Mark. You didn’t know how to work around them, and it showed.

“I’ll bring you some clothes to get changed into,” Taeyong announced before leaving the room to get you a change of clothes. You thanked him, but he was already out of earshot. After you got changed into some sweats and another sweater of Taeyong, you guys shared pizza and watched a movie. There was a weird atmosphere that no one knew how to break, so you quickly made your way toward Mark’s room to go to sleep. 

It was strange sleeping in Mark’s bed in a house filled with boys. It was weird knowing there was a corpse of a cat buried in their garden that you had accidentally killed. However, you did sleep the full ten hours. It was Mark who had woken you up to check up on you.

“Y/N,” Mark whispered while he gently touched your forehead. You opened your eyes and were met with Mark looking worriedly at you. You noticed you were covered in sweat.

“Hi.” You croaked out, noticing your dry throat. 

“Here is some water.” Mark offered you, and you gladly accepted. “You had a bad dream. Do you remember anything?” He asked you after you drowned the whole glass. You shook your head. This wasn’t the first time you were covered in sweat when you woke up. 

“It happens from time to time. I never remember the bad dream, though.” You explained to him, but he kept looking at you as if you were a porcelain doll. “You don’t need to worry. I’m a big girl.” You joked while you showed him your imaginary muscles. This got a little chuckle out of him, but not his full contagious laugh that you loved so much.

“I have to go to school now, so I can’t drive you. Taeyong told me he has the same class as you at ten, so he will be driving you, okay?” Mark explained to you, and you could only nod your head in agreement. He quickly enveloped you in a big bro hug before leaving you in his room. 

You were going to be alone with Taeyong. You would enter your class with Taeyong by your side. All the girls will be jealous, and your friends are definitely going to want to know the story behind your entrance. You couldn’t tell them you killed a cat by accident, and called Mark while having a panic attack. Perhaps you should just tell them it was all coincidental. 

“We leave in ten minutes,” Taeyong said while knocking on Mark’s door. You realized you had stayed frozen on Mark’s bed for over an hour, and you didn’t even start to get ready.

“Let’s make it twenty.” You shouted before rushing to the bathroom to take a quick shower. You heard Taeyong laugh, and you hated the fact that you always missed it when he was laughing. For someone you’ve had spied for over a year, you had never seen him smile. Now, in the span of barely twelve hours, you had heard him laugh and seen him smile. It gave you a rush of adrenaline and a certain kind of proudness that you were able to evoke these emotions out of Taeyong. Is this how people get addicted? Is this the reason why girls keep throwing themselves at him?

You didn’t have any spare clothes, so you kept wearing Taeyong’s black hoodie and your black leggings from the day before. It wasn’t your best outfit, but it wasn’t that different than other days. When you walked out to Taeyong’s car, you did feel a little bit insecure. 

“You look great.” He softly complimented you while you seated yourself next to him. A soft blush found its way on your face, and you tried to hide it while covering your face.

“You’re only saying that because I’m wearing your hoodie.” You teased him, stunned by your own sudden confidence boost. He looked just as stunned as you at your remark, but he quickly recovered, showing off his tiny gummy smile.

“You got me. You look amazing in my clothes. You should keep the hoodie.” He told you while staring the car. You stared at his hands while he drove onto the street. Your heart started beating faster, and your palms began to sweat. 

“Thank you! I think I will.” You finally muttered when you had recovered just a little bit. If he noticed your awkward behaviour, he didn’t comment on it. But you had a feeling he knew he could make your heart lose its healthy beat. 

The moment the two of you entered your local college together, everyone stopped talking so they could look at the strange combination. Taeyong was never seen with anyone except for his three friends who were coincidentally his roommates. You understood their reaction because you would act the same if he walked with any other girl into class. You thought he would sit in his usual lonely seat at the back, but he followed you toward your fixed spot at the front. You liked to sit at the front with two of your friends. You looked questionably at Taeyong, but he simply shrugged his shoulders.

“I just want to make sure you’re okay.” He softly whispered as if understanding the question in your head, moving a bit closer to you than was sociably acceptable. His closeness made you shiver, but you tried to cover it by pretending to be cold.

“I am.” You confidently stated while you tried to ignore the rush in your body by Taeyong’s closeness. 

“You’re still pale.” He remarked, again with the same worry in his eyes as the day before. You didn’t know the guy, why the heck was he so interested in you? Maybe it was because you were close to Mark, but it still confused you. You hadn’t really spoken to Taeyong before yesterday, and now he was sitting next to you in class?

“I’m always pale.” You deadpanned while taking your seat. Taeyong quickly followed and occupied the chair next to you. He softly opened up his backpack to get his notebook and pencil while you eyed him suspiciously. 

“I really don’t like this class.” He whispered, and you could feel the stare of your friends on your back. Why was the Taeyong whispering something in your ear? You ignored him for the remainder of the lesson, as his presence was already distracting enough. 

After class your parted ways, you had an economics class, while he had to do something else you didn’t ask about. Your friends fired question after question the moment Taeyong was out of earshot, but you had a feeling he could hear everything they said by the small smile on his face. You tried to answer as honestly as possible, but you mainly kept up the facade of it being an accident. That both of you run into each other on accident. They seemed to believe you, and you continued your day as if nothing had ever happened. 

You walked your usual route back home while it was already getting dark out. You had stayed longer at the library than planned, but it didn’t matter that much to you. You liked the colours of the sky when the sun was setting, reminding you of carefree times you spent your holidays on your grand parent’s farm. 

You hadn’t noticed a figure following you, and you also hadn’t felt the air getting colder and colder the more you got in the vampire district. You hugged yourself trying to get a bit warmer, but the cold was creeping in your bones. You looked at your phone, but there were no new messages, making you feel alone. 

“Why is a beautiful girl like yourself walking all alone at night?” Someone commented, and you quickly turned around toward the source of the voice. It was a handsome young man you had never seen before. Your town really was filled with beautiful young men with mesmerizing eyes. You noticed their eyes to be a reddish colour, and it did scare you, but not enough.

“It’s barely night time. The sun is still in the sky.” You corrected the stranger as if you were old friends. He seemed surprised by your answer as if he hadn’t expected a bratty response.

“I see why they like you.” He simply stated, making you look at him with confusion. Who was he talking about? He started laughing, and his laugh was low and a bit scary. You know this cliché villain laughs, well he had one like that. 

“I” m Jaehyun.” He introduced himself, walking closer to you. You took a few steps back, but he simply smirked at your actions.

“Okay.” You answered him while your back was against the corner of the street. There was no one to see you. You were alone with this stranger, who you felt had bad intentions. 

“What’s your name?” He asked you softly, still making his way closer to you. He was only an arm’s length away from you, and you didn’t know how to get out of this situation. 

“I don’t see why that’s important.” You told him, but this must have been the wrong answer. His smile disappeared, and annoyance transformed his face into something grim. Something you didn’t want to witness. He moved closer until he was right in front of you, and you could smell his evil intentions. You had never seen this man before, and it scared you. His hand softly touched your face while he sniffed the side of your neck. His lips barely touching your sensitive skin. You stood there frozen, not knowing how to handle the situation, while you felt his teeth scraping at your skin. 

You thought this was going to be the end. You knew the guy must be a vampire, so you didn’t try to run as it was useless. They were faster and all-round better at anything than humans. You were scared, and you expected his bite any second when he was forcefully removed from you. You didn’t dare to open your eyes, but you heard someone crying in pain. 

The moment you opened your eyes, you couldn’t understand what was going on before you. Taeyong was moving incredibly fast; you could barely follow him with your eyes, but he seemed to be the one with the upper hand. The Jaehyun guy tried to fight back, but Taeyong was filled with rage the other guy barely didn’t have a chance. 

“Taeyong!” You yelled, trying to stop him from killing the other. He held Jaehyun in his arms, ready to snap his neck, but he stopped when he heard the desperation in his voice.

“Do you have to kill him?” You asked him softly, looking into his eyes. He looked puzzled by your question.

“He’s been biting humans around here even though he knows there is a truce.” He told you a matter of factly, but you weren’t satisfied with his answer.

“Are you alone, Jaehyun?” You asked the guy Taeyong was holding firmly in his deadly grip. You didn’t know why you were trying to save the scary vampire’s life, but you already killed a cat this week. You couldn’t handle any more blood on your hands. 

“Why does it matter? You want to adopt me?” He spat back at you, and you almost wanted to turn around and let Taeyong do whatever he was planning to do. 

“Isn’t that better than dying?” You asked him softly. You touched his face, mimicking the action he had done to you earlier. His were still franticly searching for a way out, and you could feel he was a lost cause. But you didn’t want to give up on him so easily. There was a glimmer of desperation in his eyes as if he wanted to be killed. As if he knew someone was out there protecting you. 

“I’ve been dead for over twenty years now.” nHe simply stated, his body relaxing a bit as if he was showing his true colours. 

“You’re relatively young.” You remarked. Your eyes locked with Taeyong’s and he seemed to understand your plan. Somehow he had a perfect instinct on how you felt. It comforted you to know he actually listened to you and respected your opinion even though he didn’t really seem to agree.

“Don’t tell me you plan to take him home?” He asked you in disbelief, but he already knew the answer.

“Not my home.” You shrugged, looking over at his car that was still parked in the middle of the empty road. 

“Get in the car.” He commanded, and you followed his order as if your life depended on it. Taeyong locked the unknown vampire in his trunk as if he was used to tying up depressed vampires. The ride back to his home was quiet and filled with a thick tension you didn’t want to break. He looked deep in thought, and you couldn’t take your eyes off of him. 

“I remember a year ago where I told you not to talk to strangers.” He finally spoke while he parked his car in front of his house. You didn’t know how to answer him, so you just got out of the vehicle.

“Aren’t you scared?” He asked you forcefully. He was annoyed by your behaviour, but you weren’t scared. You had never been scared of vampires, and you could never be frightened of Taeyong. 

“Should I be scared?” You asked him softly while you looked into his eyes. He seemed to soften a little, but he still looked pissed at your behaviour.

“Yes! He tried to kill you.” He stated, and you simply shrugged your shoulders at his comment.

“I don’t think he actually tried to kill me. He was waiting for you to tear him away from me.” You explained to Taeyong while Mark went outside to inspect the noise you two were making.

“Y/N? What happened? What are you doing here?” Mark asked you immediately when he saw you. He inspected you from head to toe to see if anything was off, but you looked perfectly fine.

“Your friend has adopted a vampire.” Taeyong simply explained to mark while his eyes bore into yours. 

“What?” Mark questioned him again as if he hadn’t heard him correctly.

“You heard me. We’ll set him up in the basement. Tell the others.” Taeyong commanded Mark while he grabbed your arm and pushed you inside the house. “And you, you stay in the kitchen.” He told you when you were safely inside the house. Mark went back inside, yelling for the others to come down, while Taeyong grabbed Jaehyun from his trunk. 

You listened to Taeyong and waited in the kitchen. You had no clue what the others were doing in the basement, but you didn’t hear any noises to make you feel worried. Mark was the first to come up, but he seemed as cheerful as ever. 

“Taeyong is going to drive you home. I’m staying with Jaehyun.” Mark told you while you were scrolling through your phone. You were glad to spend some more time with Taeyong, but at the same time, it made you feel worried. Was he going to scold you?

“He’s not pissed. Don’t worry.” Mark explained when he saw your worried expression. You relaxed a little bit but still acted stiff. The past 24 hours had been tiring, to say the least.

“I’ve brought enough trouble into this house.” You confessed while you looked down at your hands. Mark quickly came over to your side to wrap you into a big bro hug.

“You’re a troublemaker, I’m not going to deny that.” He whispered with a teasing smile which you gladly returned.

“Are you trying to make me feel better?” You asked him with a raised eyebrow, earning a wholesome laugh out of him.

“I don’t know how to handle this situation.” He confessed finally.

“I’m not scared of you, and I would like to keep my best friend.” You told him earnestly. He looked relieved by your answer.

“You’re never going to lose me.” He told you. His arm was still around you when Taeyong returned to the kitchen. He had his car keys in hand, signalling you to move. You said your goodbye to Mark and followed Taeyong outside. 

The moment you stepped into his car, a strange awkwardness filled the small space. Your heart was racing, and not because you were scared, but because Taeyong was near.

“Why aren’t you scared? You should be scared of me. I could tear you apart in a second.” He finally broke the silence when you were almost at your house.

“Do you want me to be scared of you?” You asked him while fidgeting with your hands. You seemed to do that a lot around him.

“I’m no good.” He confessed softly. You looked over at him, and he looked like a broken man. He probably had never chosen this life, and it made you think that Jaehyun and Mark also seemed like they didn’t want to be like this. It made you feel sad, and you tried to comfort him, but you didn’t know how to. He pulled over in front of your parent’s house, but you didn’t want to get out just yet. You finally had him close to you, and you weren’t letting him go that easily.

“My parents warned me about drugs and freaks in the streets, but never the ones with the hazel eyes and the missing heartbeats.” As you looked into his eyes, you could feel him searching deep into your soul. “I’m not scared, because you have never given me a reason to. You were there when I needed you most.” You explained to him while he kept his eyes fixed on you. His brown eyes showed you the admiration he had for you, it made you feel dizzy and bold. You cupped his face into your hands. He let it happen as if he had anticipated your behaviour.

“You’re a special one, aren’t you?” He said while his hand found their way to your face. He gently stroked your cheek, as if he was afraid to scare you away. He looked at you as if you were a porcelain doll, and you didn’t mind. He had never looked at any of the other girls in your town. He had never even given them a second glance, and here he was comfortably stroking your face in his car. Here you were with a Taeyong who looked at you as if you were extraordinary beautiful. Like you were important.

“Only for you.” You whispered, but your speeding heart gave you away. He must have heard the change of pace because he suddenly broke out into an adorable smile.

“What are you doing Friday?” He asked you, and you couldn’t hide your excitement.

“I’m free.” You told him eagerly. “I have absolutely no plans.” You continued, which made Taeyong laugh that soft giggle of his. You had a feeling you were the only human who had ever heard him laugh like this.

“Do you maybe want to go to a movie or something?” He stuttered softly. If he weren’t a vampire, he probably would’ve been blushing. You tried to calm yourself down, but with butterflies in your stomach and a head buzzing with possibilities, there was no way you could control your speeding heart.

“Are you asking me out on a date?” You asked him teasingly, masking the sound of your heart leaping out of your chest.

“If that’s okay with you.” He asked for your permission, which almost made your heart combust into a million pieces. Could your heart survive a date with Taeyong? You weren’t so sure.

“Absolutely.” You told him. You quickly gave him a kiss on his cheek before running inside. Your action might have been bold, but you regretted your straightforwardness. It was clear Taeyong had never asked anyone out on a date, and maybe you shouldn’t have tried to push his buttons. You sneaked a glance outside, and Taeyong was still parked in front of your house. You saw him touching his cheek with a soft smile, before driving off. You were almost sure you wouldn’t survive a date with Taeyong.

Nino and the Miraculous

What if Nino got a Miraculous other than the Turtle? What kind of things might he have gotten up to with the Fox, or the Peacock, or the Bee?

Three drabbles ft. Nino and different Miraculous.

(AO3) (FF.net)

Peacock

Nino didn't really dress up on a regular basis. Anything fancier than a t-shirt and jeans made him uncomfortable and just got in his way besides. He wasn't much for jewelry, either, just a few simple rubber-band bracelets.

So having a Miraculous that came with a ridiculously elaborate outfit was...discomfiting, to say the least.

"I keep asking my kwami to make it a bit simpler, but he insisted," Paon grumbled to Chat Noir as they scanned the city for the latest akuma. The akumatized woman had gone into hiding after only a couple minutes of battle, and Ladybug was out being as visible as possible while the two boys lay hidden, waiting for the akuma to reappear and go after Ladybug so they could attack and take him by surprise. Paon shifted, tugging on the sparky robe that spread out around him and trying to hide the most glittery bits. "He said that peacocks don't do simple. And then I transformed, and he had added even more detailing to the eyes at the bottom of my robe thing. And more glitter, of course."

Chat Noir snickered at that. "Are you sure that wasn't in response to you putting the pin on in a strange spot? I swear, you have your Miraculous somewhere different every time I see you."

Paon groaned and made a face. "Maybe. But that wasn't my fault, really. It's gotten too warm to wear a jacket, so I've been pinning my Miraculous to an undershirt instead, since there's no way I could get away with pinning it to my actual shirt. It worked okay, but then I had a slightly tighter shirt and if I didn't want the outline showing to everyone and anyone then I had to pin it on my stomach. And I forgot to move it before I transformed." When Chat Noir continued snickering, Paon scowled at him. "Okay, some of us weren't lucky enough to get a Miraculous that can stay on in the same place 24/7. I would love to have a bracelet or ring instead of a pin, but I didn't exactly get to choose."

"I know, I know," Chat Noir insisted, still looking far too amused at Paon's plight. "But you have to admit, it does look funny when where your cape thingy is pinned changes every single day. Like, one day it's pinned at your right shoulder, then on your left, then in the middle, then halfway down your side, then on your stomach like some kind of weird skirt..."

Paon groaned again and flopped back on the rooftop, now sorely regretting the fact that Chat Noir was no longer avoiding him since they had established that Paon didn't have any feathers in his costume and therefore Chat Noir wasn't allergic to him. He had been hurt initially when the famous cat superhero kept his distance, but if Chat Noir was just going to tease...

"I'm gonna ask my kwami to add feathers to this glittery mess if you don't stop laughing," Paon warned after another few seconds of listening to the snickering. "Proper feathers. And then we'll see how smug you are when you're sneezing every few seconds."

Chat Noir promptly fell silent.

(Apparently Duusu had heard Paon's threat to Chat Noir, because every time he transformed for the next week, a whole down comforter's worth of feathers flowing out under and past his robe-slash-cape. Chat Noir glared at him through four fights, and it wasn't until Ladybug scolded him for taking a joke too far that he talked to his kwami and Duusu only agreed to change the costume back after several large bags of sugar-coated jelly candies. It took a month more after that for Chat Noir to agree to get anywhere near Paon again.

And Nino never could get Duusu to dial back on the costume's dramatics. He did, however, wise up and stop complaining about it before he could get completely drowned in glitter.)

  Fox

Nino was a bit bemused when he first found the fox-tail necklace in his room. It didn't see like the kind of jewelry that he would normally wear or get as a gift, but once Trixx popped up and explained things a bit (after Nino's screaming subsided, of course) it made more sense.

He hadn't been chosen to match the jewelry itself, but rather the powers that came with it. And if he thought about it, the necklace itself would be easy to keep secret. It wouldn't be hard to tuck the necklace into his shirt and hide it there when he wasn't transformed.

Besides, who cared about who the Miraculous itself looked like when the powers that came with is were so cool?

"So I can make illusions that act and sound just like the real thing?" Nino questioned Trixx once again. When the kwami nodded yet again, Nino grinned, excitement coursing through him.

Never mind generating random items and fixing-everything power. Never mind Cataclysm, even. Illusions of anything that could act just like whatever they were was so. much. cooler. It was like the cutting-edge holograms that XY used instead of performing live, except able to go whatever and say whatever- and speaking of XY, that gave him an idea.

A beautiful, fabulous idea.

"So I could make an illusion of Jagged Stone and make it look like he's playing live in my room?" Nino wanted to know. "Because, dude, that would be so. cool. My speakers are good and all, but nothing beats the real thing-"

"That is not the intended use of the Miraculous!" Trixx scolded. "You'll get yourself discovered that way! If anyone comes to investigate the noise and finds the Fox superhero sitting in your room-!"

"I could use an illusion to make myself look normal!" Nino protested, but he could already tell that Trixx made a good point. If anyone came in and touched any of the illusions and they went up in smoke like Trixx had said they were apt to do (and there was no way to deny that that wouldn't happen, not with a nosy little cousin that visited without warning, there would be no way for Nino to believably argue his way out. He would have to reserve use of his powers for battles, and battles alone.

(...well, okay, for battles and for late-night pranks, but only because even if Nino wasn't exactly much of a prankster, he just couldn't resist the opportunity.)

"Besides, one large, complicated illusion like that, and your transformation would wear off after five minutes and the illusion would go with it," Trixx finished, crushing Nino's dreams of personal hologram concerts with finality. "It's not worth it. But you say that there is technology these days that acts similarly to the Illusion? How does it work? That's so cool you can do it without magic!"

"I don't know, dude," Nino said, shrugging. "I'm not great at the science and electronics stuff like that. But I can ask one of my friends at school!" he added quickly when Trixx drooped in disappointment. "My friend Max would probably know, he's crazy smart about technology stuff. Whether or not either of us will be able to follow what he says I don't know, but I'll try."

Trixx considered Nino for a few more seconds. Then a toothy smile split his face.

"Great! You know, I think you'll do."

  Bee

Nino didn't know how the designs for the various Miraculous were chosen. If he had been in charge, he would have chosen a bracelet, definitely, and a ring, and maybe an anklet, and he supposed that maybe earrings could work for both male and female holders, and a necklace might, too-

-but a comb?

Pollen seemed just as puzzled as Nino as to how he was expected to wear the Miraculous. Apparently most of her previous holders had been female.

"And the males holders all had long hair," Pollen grumbled, inspecting Nino's close-cut style. "It was a thing back then."

Nino raised an eyebrow as Pollen unsuccessfully tried to shove the comb into his hair. The tines scraped against his scalp briefly before coming loose again, sending the kwami and Miraculous tumbling through the air. "Men wore fancy combs like yours as well?"

"No- well, not normally," Pollen said, stabbing at his hair again. It wasn't any more successful than the first time. Nino hid a wince as the comb scraped against his scalp. "But my Chosens had lovely thick hair and would hide my comb in their hair. Some hid it in their beards. Can you grow a beard?"

"How old do I look?" Nino questioned, exasperated. "No, I can't grow a beard, it's impossible."

Pollen buzzed in frustration and flew a lap around his head. "Oh, I don't know what else to do! You could stick it under your hat, I guess, but that would come off too easily- and what if your hat blew off and someone saw it? Or what if you had to take off your hat inside? You're always adjusting it, it wouldn't be safe."

"I could stick it in my pocket," Nino offered, reaching out to take the comb. Pollen snatched it away, scowling. "Come on, there really aren't that many other options. My pockets are deep enough to handle it, and I always keep my phone in the other pocket."

"But then it'll look weird when you transform," Pollen complained. "Not that it would look any less funny if we just glued the comb to your head. And you would probably complain about it not feeling great then, too."

Nino cringed at the thought. "We are not gluing anything to my head. It would look weird under my hat, anyway. And I have to take my hat off on a regular basis, too." He considered the Miraculous again. "I'm going to have to have it in my pocket. We'll have to see how it looks when I transform, I guess." He moved to tuck the Miraculous in his pocket, tugging it away from Pollen. "We can at least try it this way."

"Wait, what's that?" Pollen asked as Nino's shirt hitched up to reveal his belt buckle. "That metal thing there, on your pants. With the leather."

"That's my belt," Nino said, puzzled. He tugged his shirt up a little more so Pollen could see it without diving under his shirt to investigate. Kwamis apparently had absolutely no sense of personal space. "I don't always wear it, but these pants are a little big and I wouldn't want them to start slipping or anything if I have to run from an akuma attack."

Pollen spun in circles, excited. "This is perfect! You can glue the Miraculous to your belt! It'll look a bit strange when you transform, but it'll be hidden by your shirt the rest of the time."

Nino glanced at the Miraculous, then at his plain belt buckle. "Really? You think that will work? I feel like that could get uncomfortable too, if I need to bend over or something."

Pollen level him with a steely look. "It's either getting attached to your belt or your head. Your choice."

"...I think I'll go with the belt."

-0-0-0-

Chat Noir circled his and Ladybug's new teammate, inspecting his outfit. It looked fairly standard for a superhero outfit...except for the comb held by a black-and-yellow strap at hip height.

"Interesting fashion choice," Chat Noir commented, grinning. "Starting a new trend?"

Bumblebee groaned. "There was literally nowhere else I could put it," he explained. "My hair is too short and Pollen wouldn't let me just stick the comb in my pocket. I'll deal with the funky look."

Ladybug looked like she was trying not to laugh. "You must have a pretty big belt buckle normally, then."

"Oh, you have no idea. My kwami seems pleased, though, so..." He shrugged. "I guess I'll wear long shirts until ol' Hawky gets defeated. Won't be a problem unless I hit a growth spurt or something."

Chat Noir laughed but finally quit his prowling to lean against a nearby chimney. "That's a funny superhero requirement. No growing allowed."

"Says the one who's shot up several centimeters in the past month alone, you beanpole." Ladybug's teasing tone held no malice. "I'm glad you found a solution, though," she added to Nino. "As unusual-looking as it may be. And Bumblebee?"

"Yeah?"

"Welcome to the team."

REALLY  LONG  CHARACTER  SURVEY.

RULES.   repost,   don’t  reblog,   tag 10,   good luck! TAGGED.   nobody.  stolen from @justpawns​ TAGGING.   whoever.

BASICS.

FULL  NAME :   elizabeth marie spencer /  elisheva bat martha NICKNAME :   liz AGE :   25 BIRTHDAY :   june 25th LANGUAGE / S :   english, french, spanish, some german SEXUAL  ORIENTATION :   bisexual. ROMANTIC  ORIENTATION :   grey romantic RELATIONSHIP  STATUS :   single, though verse dependent CLASS :   middle HOME  TOWN / AREA :   shenandoah in virginia  CURRENT  HOME :   nyc (though this is verse dependent) PROFESSION :    associate information systems officer for the statistic division of the united nations (it specialist for any company within verse dependent city)

PHYSICAL.

HAIR :   near platinum blonde with golden highlights. normally, the length of her hair is just past her collarbone, she does wear hair extensions for fancier up-does or being able to flip her hair for high ponytails. she wears a lot of buns for work, then messier buns on lazy weekends. naturally, her hair is a lighter brown. EYES :   light brown, sometimes appearing honeyed. FACE :   tbh i am the worst at describing face shapes. but hers appears a little long (or ig oval?) with somewhat prominent cheekbones that are more noticeable with contour. LIPS :   pink & thin. her bottom lip is much fuller than her top. sometimes, the edges of her top lip disappears at certain expression. she curls her lips into her mouth a lot, where you can’t see them. she applies lipstick (business, formal, seduction) & lipgloss (casaul lipcare). COMPLEXION :   pale, but not overally pale (i sound like i’m writing my immortal here haha). is smooth but naturally dry. BLEMISHES :   while h.alston does have a beauty mark on her cheek, liz would not have one. her nose is prone to blackheads when she skips facial care. she has a birthmark under her right shoulderblade that appears like a spotty star. SCARS :   a medium scar at the left side of her head from the time  TATTOOS :   none, though can be different base on private verses HEIGHT :   5′2″. WEIGHT :   105 lbs, though she can fluctuate from 99 to 110. it very fickle base on her stress levels. BUILD :   at average, she’s barely a healthy weight. the higher the stress the lower the weight. she’s very petite.     FEATURES :   she has a sharpness about her that she uses to cut at her advantage. ALLERGIES :   n / a. USUAL  HAIR  STYLE :   it’s a different style everyday. thought mentioned in the hair section, her usually style are buns, either more formal or messier with loose ends. USUAL  EXPRESSION :   deadpan. while she tries to make herself hard to read, there’s either constant irritation or a hint of mischief.  USUAL  CLOTHING :    i’m linking her fashion board because it’s so vast & beyond my capabilities.

PSYCHOLOGY.

FEAR / S :   closed spaces, lost of control ASPIRATION / S :   success. power. if i have to sum them up. but also protect who she consider underdogs (re: her siblings).   POSITIVE  TRAITS :   loyal,    level-headed,    cautious,    trustworthy. NEGATIVE  TRAITS :   selfish MBTI :   estj-a

Extraverted  60%        40% Introverted  Intuitive 32%        68% Observant  Thinking 72%        28% Feeling   Judging 88%    12% Prospecting   Assertive 58%        42% Turbulent  

ZODIAC :   cancer. TEMPEREMENT :   melancholic. ANIMALS :   she relates to horses, but she acts like a lizard cat. FAITH :   athiest. GHOSTS ? :   no. AFTERLIFE ? :   no. REINCARNATION ? :   she’s top tier what do you mean she doesn’t care ALIENS ? :   no. she’ll pull out mathematical equations as to why, even if they exist, it’s impossible to know. :/ EDUCATION  LEVEL :   master. (for au’s with her father dead, she will get a phd to become a professor in his honor, since he always wanted her to be a teacher).

FAMILY.

FATHER :   robert spencer : superintendent of rockingbridge county’s schools. liz’s role model, though he’s even able to hide just how cold he can be to people. liz is also clearly his favorite kid. while he appears to have a big heart (re : free ice cream to a grade when he visits the elementary schools when he was a principal), he’s pretty deadpan otherwise & does certain deeds for status. especially to his wife, who he hardly talks to. MOTHER :   martha spencer : vice principal of middle school. fairly stern like, all the time, but does so in care. while her & liz have more in common personality wise, this often clashes & the two don’t get along well. robert’s favoritism doesn’t help either, though she was already having favoritism over raleigh. early on, she would lash out at liz if she was mad at robert (who was often ignoring her), but changed that as liz grew older. SIBLINGS :   raleigh spencer (older brother) : he’s a doctor settled in nyc, living kinda close to liz. but they look & act complete opposite. he gets along with everyone, though he’s probably too trusting at times. does a lot of volunteer work, bikes, & goes to cons on his free time. hasn’t liz that (of course) he actually makes more money than liz, since they always had a mini rivalry (enforced by their parents). as said, they use to not get along, but now are pretty close.  madeline ‘maddie’ spencer (younger sister) : currently a music major at jmu. she was forced to go to college by her parents (and liz, though she was more like ‘try it, & if it’s not for you then drop out’ sorta talk). while she likes college so far, she feels it’s because it’s a way to get away from her parents’ increase fights (and regrets leaving raphine, with the last spencer’s departure bring the fights to their apex) & her high school best friend is super supportive. while her goal is to be a rockstar, she’ll end up doubling major in music & education & become an elementary music teacher (and be the best at it). she also keeps telling liz she’s going to live with her at nyc because maddie looks up to liz. after all, with feeling like the outcast child for her parents, maddie jokes that liz raised her (to the point liz is ‘mom’ in maddie’s phone). EXTENDED  FAMILY :   more than my fingers & toes. like, on her dad’s side, robert had five siblings (four are still living), with one of them having seven kids; the other, including the deceased raleigh, has at least one child. heck, even liz’s ex married into the family. then, her mom’s side has a few relatives in nova (near dc) & boston. martha has three siblings: two in nova & one in boston with some cousins. like i said, it’s huge. NAME  MEANING / S :   ‘oath of god’   HISTORICAL  CONNECTION ? :   it’s her grandmother’s name, & her middle name is her grandmother’s middle name as well. in jewish tradition, its common to name offspring after dead relatives for their honor, & grandmother elizabeth died while martha was pregnant with liz, so the name felt befitting / that’s what robert wanted anyways. but  elisheva is the historical name of aaron’s wife from the torah.

FAVOURITES.

HOLIDAY :   she doesn’t have a favorite holiday, but her least favorite is christmas MONTH :   june. SEASON :   summer. PLACE :   a horse trail, either in the mountains or on a beach WEATHER :   a summer sunrise. SOUND :   brewing coffee. SCENT / S :   coffee. wine. clean linens. TASTE / S :   coffee. wine. apples. chocolate. FEEL / S :  silk. ANIMAL / S :   horse NUMBER :   3 COLOUR :   navy.

EXTRA.

TALENTS :   math stuff. coding.  BAD  AT :  art. literacy.  TURN  ONS :   flustering partner. praising. glasses. TURN  OFFS :    domination over her. aggression. HOBBIES :   equestrian. reading. cooking. overworking.

FC INFO.

MAIN  FC / S :   ha.lston sage ALT  FC / S :   none, but ca.rlson y.oung is her bodyclaim OLDER  FC / S :   na. YOUNGER  FC / S :   na. VOICE  CLAIM / S :   emma stone 

MUN QUESTIONS.  Q1 :   if  you  could  write  your  character  your  way  in  their  own  movie ,   what  would  it  be  called ,  what  style  would  it  be  filmed  in ,  and  what  would  it  be  about ?          A1 :    well, ironically, the movie would not be about them. liz is a side character. like, her main thing is being the wife of the main character in a distopia thingy (remember destiny?). however, if being the main in a movie, it would either be a romcom or a teenage drama. maybe a star wars spin off if you count the verse with sophie & charlie (and, then, eliza is the main character duh).

 Q2 :   what  would  their  soundtrack / score  sound  like ?            A2 :   pop folk music... does mumford & sons count?

 Q3 :   why  did  you  start  writing  this  character ?            A3 :   as a hetalia oc because i was having a bad time writing a canon character. and nobody was making a virginia & it’s my homestate so... yeah. it’s embarrassing.

 Q4 :   what  first  attracted  you  to  this  character ?  A4:     her crazy antic is fun to write. she’s so different from me that it helps getting into mindsets of characters i don’t relate to until i have something to relate to.

 Q5 :   describe  the  biggest  thing  you  dislike  about  your  muse.        A5 :    she’s so snobbish. like her interests (fashion) is beyond my expertise & it’s hard to add into writing when i can’t tell black & navy apart.

 Q6 :   what  do  you  have  in  common  with  your  muse ?            A6 :   the massive family dynamic & being an outlier of it. namely liz’s political opinion with her dad’s side of the family versus my opinions with my entire family. also, how we don’t really care about romance, though i’m a bit more open about it happen when it happens while she’s being against it.

 Q7 :   how  does  your  muse  feel  about  you ?          A7 :   she wouldn’t know i exist. and i would be scared of her

 Q8 :   what  characters  does  your  muse  have  interesting  interactions  with ?          A8 :   when she’s most uncomfortable, like around dogs & kids.

 Q9 :   what  gives  you  inspiration  to  write  your  muse ?          A9 :   ta.ylor sw.ift songs. music, definitely. i have a playlist of songs to help.

 Q10 :   how  long  did  this  take  you  to  complete ?            Q10 :   it over two days? because i get distracted & my migraine last night didn’t help. i work slow to begin with.

Korean Famous Musician Thingy

A/N: I know you waited long for this @channynum ! Hope you like how it turned out! And thank you so much for your cheerful words! <3 (gif and images aren’t mine) This is only my opinion. Please don’t be sad if this doesn’t turn out like you hoped it would! 

 Musician

Your Company:

Position:  Main-Dancer, Rapper “Nicknames”: Mum, Brainy Years as trainee: 2 Group: 5 member co-ed group (2 boys and 3 girls) Debut: 2013 Songs: The songs are mostly more on the mainstream side, with the focus on the choreography and crazy MV’s. Besides that, your group has also a really strong rap-line which means some songs even come closer to hip-hop than pop. Being around for quite some time your groups had tried a loud of concepts, but no school-girl/boy one yet because it’s more the “cool and crazy” concept that your group likes. But from MV’s that just make no sense and are just fun to watch it goes also to really meaningful stuff, where fans create thousands of theories and storylines for the MV’s. You’re MV’s get around 60M views, but the one with the most even has 170M views. Comeback: This time your group decided to try in a completely new concept. Going away from the mainstream path the songs has a deep meaning with an artistic dance and is more of a ballad. Similar to:

Fans love how the song hits right into the feels, especially if you consider the group's backstory. Bc like for many groups it has been really hard at first. Also, the song talks about how thought times were a lot easier because of all the loyal fans that stayed at the group's side and supported them. The songs are written and composed by all the members what is a wonderful present again. Especially the rap parts are really emotional. This is definitely the most personal comeback by your group.

History: You were a trainee at JYP but then debuted with a group from pledis entertainment. That was kind of hard for you at first because you trained with different people and now suddenly you were introduced to new trainees that will be your group. But despite your fears of not getting along with the members, because you are new, they quickly accepted you. Soon you and the other members were getting along just fine. Your group is really popular and there for some time now. There have been hard times at first but now it’s easier and many people appreciate your talents. You: You have quickly become a really important part of the group. The other’s often come to you when they have problems with learning a choreography because you pick the movements up the fastest. While you’re the best dancer you still needed some help with your rapping skills, therefore you often get help from the others. Solo career: You have released your own mixtape awhile ago and it was really successful especially the MV’s you put out for two of the songs. Now you’re contemplating on creating a mixtape 2.0 put you’re not sure yet if you have time with your busy schedule. Fans: This fandom is so freaking loyal, it’s unreal. Of course like every famous group there are also sasaengs fans, but you can’t even really count them as fans. But despite many hard times and troubles at first, the fans are always there to support your group. No matter what haters say, the fans will always prove them wrong. Also, they are really good at voting and always craving for a vlive.  And the amount of inside jokes the fandom has is remarkable. The fans are all just nice people that just find joy in your group's music and can really relate to the members. Ships: There is one ship that has been there since the beginning. It’s between you and the Main-rapper (25, male). The friendship between you tow is just really cute and you two always help each other, because he isn’t that good at dancing so you help him. A ship that has formed lately, is between you and the Maknae (20, female, Vocalist). You two are really good friends and many fans say there is more. But we will never know. Still, one thing is very clear you two are a dream team. Rumours: So many dating scandals. No matter if with other idols or in the group, every time the public sees one of the members going for dinner with someone, the dating rumours are starting. After some time it really gets annoying and neither the company nor your group says anything to it. So the rumours are never denied nor confirmed. Besides that, there has also been a drug scandal with one of the older members, but it turned out that it had only been fake news by haters. Still, around that time it had been really hard for the group. Idol Friendships:

Bang Chan (Stray Kids)

Min Yoongi aka Suga (BTS)

Momo Hirai (Twice)

Implementing Minimalism!

I have done so much since the last post!! My new thing is decluttering. I am trying to downsize by 80-90%, which is actually a TON of work. Going through and trashing everything /sounds/ easy, but it's actually really physically exhausting. I think I'm finally building up my back muscles though! 💪 So... a list of things I am working on/ have done: 1. I finally read the book about the "konmari method" of decluttering and found it pretty inspiring! I've actually pretty much followed those rules without meaning to (no wonder people kept recommending that book to me!), but in the past, I'd just chuck things if I didn't use them, joy-bringing or not, which made me sad until I forgot about it. Now, it's about bringing joy into my life in everything I own. For example, I found a vintage muffin pan at a used bookstore today. Looking at it just makes me smile!Now, I'm free to get rid of my nasty old ones and the bad memories of the giver with them. :) 2. I am working on minimizing my spice collection. I already downsized, but holy Jesus I do not need to have a basket overflowing with spices. I cook for TWO-- really, ONE now with Damion off trucking, not 12. I was having trouble picking the ones I would keep, so I picked an arbitrary number of spices I would allow myself. So, I picked 10, not including salt/pepper. Then, I asked Damion to choose the 10... because I got all, "ohhh!! Curry!! I love Indian food!! Remember that one time I had... " etc, so I was just unable to make a rational decision. He picked: *garlic salt *lemon pepper *parsley *basil *oregano *nutmeg *paprika *cinnamon *cloves *ginger I felt that this list was pretty comprehensive, so I went out and bought little magnetic tins for those suckers ($3 a piece!!) so that in the tiny house I can stick them on a strip or the fridge. For now, they will live in the junk drawer (once I eliminate it:)). 3. I started a secret garden:3 but, what's new about my gardening philosophy is that I'm only growing the stuff I'll actually use (other than my flowers:>). Most of my space is in part shade, so I'm mostly doing herbs: basil, parsley... Basically, my goal is to get everything on the herb list above growing! I also got mint and chamomile cause I love those teas. 😍 I did run into a problem here though: while in the herb section, I found rosemary and cilantro. :/ I started remembering how delicious they are, and how we do use them frequently. Then, I saw lavender, which is my herb weakness. It just smells soooo good, and the flowers are so pretty! I don't use it though, which makes it a "clutter" purchase... and that freaked me out so bad, I put back my rosemary and cilantro too and left that area immediately! They are mentally "on hold" until I have decided what to do. I know I probably won't get the lavender (until I can find a legit reason to justify it), but idk if I'll add the two useful herbs to my 10 (making 12 total) or if I'll cut off paprika and some other spice. But, chiliiiii!! And, damion didn't even list chilli pepper already! I really suck at minimizing spices, obviously. The $3/ magnetic spice tin cost does help motivate me a little, though. :) 4. I have started downsizing, so my house is easier to clean. Above all, I am a lazy person. Less dishes = less doing dishes, so I tossed them all and am now using two each of my inheritance plates, bowls and serving bowls (also used as mixing bowls). I think of my wonderful grandma, RIP, each time I see them now, and so eating gives me the feels 💖 I think about her almost every day now and it makes me happy. :> I downgraded Damion's and my clothing (Damion participated via Skype). I already had downsized into about... 3/4 large drawers with no hanging items, but they were crammed in. I thought I was done, but then I had a thought: "Katrina. You are lazy. It's going to suck when you inevitably put off your laundry and have to do it all at once!" ... So, I posted a question in the minimalist group I'm a part of and decided on 3/4 loads of laundry total. That's the most laundry in a row I am willing to do, including sheets and pillowcases, towels and washcloths. So, I got rid of everything else. :> I went through my clothes konmari style and eliminated, motivated by easier cleaning. Dammit, I want to be the type of person who washes their sheets every week! So, I got rid of stuff, separating it into selling (via consignment), donate and trash piles. I downsized to a roughly 2 week supply of stuff. Together, including linens, I only have 3.5 loads of laundry:D! Perfect for lazy Katrina, especially with my mother in law's awesome fancy big wash machine;) I now have: Linens: *8 washcloths, for face washing and exfoliation. *1 microfiber cloth, in danger of being tossed, because I HATE the texture of microfiber but appreciate its usefulness *3 towels: 1 hugeeee, and the two softest towels *1 set of fitted/flat sheets, because I won't wash them if I have more than one. *8 pillowcases: 4 normal, gray everyday pillowcases and 2 normal blue ones, for if I had a guest, idk. Plus 2 purty ones. What can I say? They bring me joy. Typing this though, maybe I should downsize more though, 8 is a bit excessive for 2 people with 4 pillows 😂 Clothing: I'll have to post about later, because my method is so involved. But, basically, everything I have matches everything else. It's all color theory and flattering fits. I plan to have excessive amounts of undies, though, because I like to change them a lot :> But, I'm only keeping 4 bras. 1 t-shirt bra, 1 convertible fancy schmancy bra, 1 sessy sessy bra ;D and 1 sports bra comfy bra thing. I love bras, but I always default to two or three anyway. My "color scheme" thing for my undies (ONLY undies, my outside clothes have a much more defined pallet) is that my bras are pretty plain/neutral, but I can get whatever panties in whatever color I want. :3 So, I am still free to indulge my love of undies and still have order :3 Last thing on implementing minimalism: I am in progress of minimizing my cleaning supply collection. I realllllly want to be earth friendly, but my cleaning stuff does not reflect that. It's all super harsh chemicals, because in the past, I was messy. I AM messy, but now my house is halfway optimized for cleaning, so it's a LOT easier for messy Katrina to keep my (her? Lol) house not clean, but within-30-minutes-of-non-rushed-cleaning clean. :D Anyways, before I had to have the harsh stuff because when I cleaned, I REALLY felt that bleach/ harsh chemicals were necessary in order to make up for the month and a half of no cleaning (wish I were joking 😂). Now that my house is easier to clean, I don't need those crazy chemicals. Except bleach, because after all, I do own chickens. I'm looking into doing my own cleaning supplies mostly: nothing crazy or complicated (I still have to cater toward lazy Katrina), but just things like vinegar, hydrogen peroxide, baking soda... etc. I still have to come up with eco friendly replacements for two things though: disinfectant wipes and the toilet wand... thingy. For the disinfectant wipes, I know that I could make my own, but it does make more laundry, and more laundry lowers the success rate of keeping my house clean. Plus, how would I keep the rags secured in sanitizing fluid without it disintegrating the rags? Or does a regular, non-bleach formula not do that? No idea. Also, I'll need to figure out how many wipes I need for a two week period, which will be tough, because I don't clean every day consistently. The toilet wand solution is harder for me though, so input would be much appreciated. I CANNOT stand the normal toilet scrubber thingies. They just don't seem sanitary, even if you keep them in cleaning liquid. Plus, they just inspire me to use copious, embarrassing amounts of straight bleach in order to sanitize, which defeats the eco friendly purpose anyway. Anyway, I discovered those toilet wands about a year ago and I haven't gone back! The wands detach from the scrubby pad, and you toss the scrubby in the trash can (because the flushable wipes suck for hard cleaning), and that's IT. You keep the nice, clean wand and get another the next time. These are also really convenient (new xD) for scrubbing my bath tub, I have discovered recently. No ridiculous hand scrubber and bleach everywhere, just this presoaked scrubber that gets everything up. It doesn't even have a chance to hurt my back, it's so fast!! Obviously, though, this is hugely irresponsible for the earth, even if they are disgusting. :( I have come up with an idea to somehow get a reusable sponge attached to the end of one of those sticks, and dropping them immediately in some dirty cleaning items container (cause they would have to be separate from clothing/linens, other than possibly disinfectant wipes) so that they could later be sanitized. But, that sounds like a lot of work, and I already hate sponges. I threw mine away in favor of those ikea scrubbing brushes :> anyway, the only way I can think to impliment this idea is to use the scrubbing bubbles version of the wand (I have both already), which has a tong-like opening for its pads. But, I don't really care for that wand. The clorox one if much more ergonomic and very sturdy, which is critical for seldom-cleaning Katrina. Also, if I'm going to have to sanitize them anyway, doesn't that defeat the whole "earth friendly" purpose? Or, could I boil them, or would that ruin the material of both the disinfectant rags / scrubbers? And, the biggest question: would I actually do it? So, yeah, basically, I need a separable toilet cleaning brush, with an ergonomic handle and a sturdy brush-to-wand attachment, with a viable, super easy way of sanitizing the brush/scrubby bit, preferably something that could be done with the disinfectant wipes too at the same time. Is that too much to ask? XD I really would love to hear suggestions though. Anyway, that's basically where I'm at right now in minimizing. I'll leave you with something a Minimalism group member said to me, to address my regrets at filling the landfills with my unused junk: either way, the earth suffers. Keeping items you don't want/need in your house doesn't save them. The only solution is being mindful about what you buy and what you support to begin with. 🌎🌳💚 This quote has helped me tremendously, as I was pretty much unknowingly using my earth-friendly side to justify holding onto items that really needed to move on. And with that, ladies and gentlemen, thank you and good night! 😘

(Via: Hacker News)

The log/event processing pipeline you can't have

Let me tell you about the still-not-defunct real-time log processing pipeline we built at my now-defunct last job. It handled logs from a large number of embedded devices that our ISP operated on behalf of residential customers. (I wrote and presented previously about some of the cool wifi diagnostics that were possible with this data set.)

Lately, I've had a surprisingly large number of conversations about logs processing pipelines. I can find probably 10+ already-funded, seemingly successful startups processing logs, and the Big Name Cloud providers all have some kind of logs thingy, but still, people are not satisfied. It's expensive and slow. And if you complain, you mostly get told that you shouldn't be using unstructured logs anyway, you should be using event streams.

That advice is not wrong, but it's incomplete.

Instead of doing a survey of the whole unhappy landscape, let's just ignore what other people suffer with and talk about what does work. You can probably find, somewhere, something similar to each of the components I'm going to talk about, but you probably can't find a single solution that combines it all with good performance and super-low latency for a reasonable price. At least, I haven't found it. I was a little surprised by this, because I didn't think we were doing anything all that innovative. Apparently I was incorrect.

The big picture

Let's get started. Here's a handy diagram of all the parts we're going to talk about:

The ISP where I worked has a bunch of embedded Linux devices (routers, firewalls, wifi access points, and so on) that we wanted to monitor. The number increased rapidly over time, but let's talk about a nice round number, like 100,000 of them. Initially there were zero, then maybe 10 in our development lab, and eventually we hit 100,000, and later there were many more than that. Whatever. Let's work with 100,000. But keep in mind that this architecture works pretty much the same with any number of devices.

(It's a "distributed system" in the sense of scalability, but it's also the simplest thing that really works for any number of devices more than a handful, which makes it different from many "distributed systems" where you could have solved the problem much more simply if you didn't care about scaling. Since our logs are coming from multiple sources, we can't make it non-distributed, but we can try to minimize the number of parts that have to deal with the extra complexity.)

Now, these are devices we were monitoring, not apps or services or containers or whatever. That means two things: we had to deal with lots of weird problems (like compiler/kernel bugs and hardware failures), and most of the software was off-the-shelf OS stuff we couldn't easily control (or didn't want to rewrite).

(Here's the good news: because embedded devices have all the problems from top to bottom, any solution that works for my masses of embedded devices will work for any other log-pipeline problem you might have. If you're lucky, you can leave out some parts.)

That means the debate about "events" vs "logs" was kind of moot. We didn't control all the parts in our system, so telling us to forget logs and use only structured events doesn't help. udhcpd produces messages the way it wants to produce messages, and that's life. Sometimes the kernel panics and prints whatever it wants to print, and that's life. Move on.

Of course, we also had our own apps, which means we could also produce our own structured events when it was relevant to our own apps. Our team had whole never-ending debates about which is better, logs or events, structured or unstructured. In fact, in a move only overfunded megacorporations can afford, we actually implemented both and ran them both for a long time.

Thus, I can now tell you the final true answer, once and for all: you want structured events in your database.

...but you need to be able to produce them from unstructured logs. And once you can do that, exactly how those structured events are produced (either from logs or directly from structured trace output) turns out to be unimportant.

But we're getting ahead of ourselves a bit. Let's take our flow diagram, one part at a time, from left to right.

Userspace and kernel messages, in a single stream

Some people who have been hacking on Linux for a while may know about /proc/kmsg: that's the file good old (pre-systemd) klogd reads kernel messages from, and pumps them to syslogd, which saves them to a file. Nowadays systemd does roughly the same thing but with more d-bus and more corrupted binary log files. Ahem. Anyway. When you run the dmesg command, it reads the same kernel messages (in a slightly different way).

What you might not know is that you can go the other direction. There's a file called /dev/kmsg (note: /dev and not /proc) which, if you write to it, produces messages into the kernel's buffer. Let's do that! For all our messages!

Wait, what? Am I crazy? Why do that?

Because we want strict sequencing of log messages between programs. And we want that even if your kernel panics.

Imagine you have, say, a TV DVR running on an embedded Linux system, and whenever you go to play a particular recorded video, the kernel panics because your chipset vendor hates you. Hypothetically. (The feeling is, hypothetically, mutual.) Ideally, you would like your logs to contain a note that the user requested the video, the video is about to start playing, we've opened the file, we're about to start streaming the file to the proprietary and very buggy (hypothetical) video decoder... boom. Panic.

What now? Well, if you're writing the log messages to disk, the joke's on you, because I bet you didn't fsync() after each one. (Once upon a time, syslogd actually did fsync() after each one. It was insanely disk-grindy and had very low throughput. Those days are gone.) Moreover, a kernel panic kills the disk driver, so you have no chance to fsync() it after the panic, unless you engage one of the more terrifying hacks like, after a panic, booting into a secondary kernel whose only job is to stream the message buffer into a file, hoping desperately that the disk driver isn't the thing that panicked, that the disk itself hasn't fried, and that even if you do manage to write to some disk blocks, they are the right ones because your filesystem data structure is reasonably intact.

(I suddenly feel a lot of pity for myself after reading that paragraph. I think I am more scars than person at this point.)

ANYWAY

The kernel log buffer is in a fixed-size memory buffer in RAM. It defaults to being kinda small (tens or hundreds of kBytes), but you can make it bigger if you want. I suggest you do so.

By itself, this won't solve your kernel panic problems, because RAM is even more volatile than disk, and you have to reboot after a kernel panic. So the RAM is gone, right?

Well, no. Sort of. Not exactly.

Once upon a time, your PC BIOS would go through all your RAM at boot time and run a memory test. I remember my ancient 386DX PC used to do this with my amazingly robust and life-changing 4MB of RAM. It took quite a while. You could press ESC to skip it if you were a valiant risk-taking rebel like myself.

Now, memory is a lot faster than it used to be, but unfortunately it has gotten bigger more quickly than it has gotten faster, especially if you disable memory caching, which you certainly must do at boot time in order to write the very specific patterns needed to see if there are any bit errors.

So... we don't do the boot-time memory test. That ended years ago. If you reboot your system, the memory mostly will contain the stuff it contained before you rebooted. The OS kernel has to know that and zero out pages as they get used. (Sometimes the kernel gets fancy and pre-zeroes some extra pages when it's not busy, so it can hand out zero pages more quickly on demand. But it always has to zero them.)

So, the pages are still around when the system reboots. What we want to happen is:

The system reboots automatically after a kernel panic. You can do this by giving your kernel a boot parameter like "panic=1", which reboots it after one second. (This is not nearly enough time for an end user to read and contemplate the panic message. That's fine, because a) on a desktop PC, X11 will have crashed in graphics mode so you can't see the panic message anyway, and b) on an embedded system there is usually no display to put the message on. End users don't care about panic messages. Our job is to reboot, ASAP, so they don't try to "help" by power cycling the device, which really does lose your memory.) (Advanced users will make it reboot after zero seconds. I think panic=0 disables the reboot feature rather than doing that, so you might have to patch the kernel. I forget. We did it, whatever it was.)

The kernel always initializes the dmesg buffer in the same spot in RAM.

The kernel notices that a previous dmesg buffer is already in that spot in RAM (because of a valid signature or checksum or whatever) and decides to append to that buffer instead of starting fresh.

In userspace, we pick up log processing where we left off. We can capture the log messages starting before (and therefore including) the panic!

And because we redirected userspace logs to the kernel message buffer, we have also preserved the exact sequence of events that led up to the panic.

If you want all this to happen, I have good news and bad news. The good news is we open sourced all our code; the bad news is it didn't get upstreamed anywhere so there are no batteries included and no documentation and it probably doesn't quite work for your use case. Sorry.

Open source code:

logos tool for sending userspace logs to /dev/klogd. (It's logs... for the OS.. and it's logical... and it brings your logs back from the dead after a reboot... get it? No? Oh well.) This includes two per-app token buckets (burst and long-term) so that an out-of-control app won't overfill the limited amount of dmesg space.

PRINTK_PERSIST patch to make Linux reuse the dmesg buffer across reboots.

Even if you don't do any of the rest of this, everybody should use PRINTK_PERSIST on every computer, virtual or physical. Seriously. It's so good.

(Note: room for improvement: it would be better if we could just redirect app stdout/stderr directly to /dev/kmsg, but that doesn't work as well as we want. First, it doesn't auto-prefix incoming messages with the app name. Second, libc functions like printf() actually write a few bytes at a time, not one message per write() call, so they would end up producing more than one dmesg entry per line. Third, /dev/kmsg doesn't support the token bucket rate control that logos does, which turns out to be essential, because sometimes apps go crazy. So we'd have to further extend the kernel API to make it work. It would be worthwhile, though, because the extra userspace process causes an unavoidable delay between when a userspace program prints something and when it actually gets into the kernel log. That delay is enough time for a kernel to panic, and the userspace message gets lost. Writing directly to /dev/kmsg would take less CPU, leave userspace latency unchanged, and ensure the message is safely written before continuing. Someday!)

(In related news, this makes all of syslogd kinda extraneous. Similarly for whatever systemd does. Why do we make everything so complicated? Just write directly to files or the kernel log buffer. It's cheap and easy.)

Uploading the logs

Next, we need to get the messages out of the kernel log buffer and into our log processing server, wherever that might be.

(Note: if we do the above trick - writing userspace messages to the kernel buffer - then we can't also use klogd to read them back into syslogd. That would create an infinite loop, and would end badly. Ask me how I know.)

So, no klogd -> syslogd -> file. Instead, we have something like syslogd -> kmsg -> uploader or app -> kmsg -> uploader.

What is a log uploader? Well, it's a thing that reads messages from the kernel kmsg buffer as they arrive, and uploads them to a server, perhaps over https. It might be almost as simple as "dmesg | curl", like my original prototype, but we can get a bit fancier:

Figure out which messages we've already uploaded (eg. from the persistent buffer before we rebooted) and don't upload those again.

Log the current wall-clock time before uploading, giving us sync points between monotonic time (/dev/kmsg logs "microseconds since boot" by default, which is very useful, but we also want to be able to correlate that with "real" time so we can match messages between related machines).

Compress the file on the way out.

Somehow authenticate with the log server.

Bonus: if the log server is unavailable because of a network partition, try to keep around the last few messages from before the partition, as well as the recent messages once the partition is restored. If the network partition was caused by the client - not too rare if you, like us, were in the business of making routers and wifi access points - you really would like to see the messages from right before the connectivity loss.

Luckily for you, we also open sourced our code for this. It's in C so it's very small and low-overhead. We never quite got the code for the "bonus" feature working quite right, though; we kinda got interrupted at the last minute.

Open source code:

loguploader C client, including an rsyslog plugin for Debian in case you don't want to use the /dev/kmsg trick.

devcert, a tool (and Debian package) which auto-generates a self signed "device certificate" wherever it's installed. The device certificate is used by a device (or VM, container, whatever) to identify itself to the log server, which can then decide how to classify and store (or reject) its logs.

One thing we unfortunately didn't get around to doing was modifying the logupload client to stream logs to the server. This is possible using HTTP POST and Chunked encoding, but our server at the time was unable to accept streaming POST requests due to (I think now fixed) infrastructure limitations.

(Note: if you write load balancing proxy servers or HTTP server frameworks, make sure they can start processing a POST request as soon as all the headers have arrived, rather than waiting for the entire blob to be complete! Then a log upload server can just stream the bytes straight to the next stage even before the whole request has finished.)

Because we lacked streaming in the client, we had to upload chunks of log periodically, which leads to a tradeoff about what makes a good upload period. We eventually settled on about 60 seconds, which ended up accounting for almost all the end-to-end latency from message generation to our monitoring console.

Most people probably think 60 seconds is not too bad. But some of the awesome people on our team managed to squeeze all the other pipeline phases down to tens of milliseconds in total. So the remaining 60 seconds (technically: anywhere from 0 to 60 seconds after a message was produced) was kinda embarrassing. Streaming live from device to server would be better.

The log receiver

So okay, we're uploading the logs from client to some kind of server. What does the server do?

This part is both the easiest and the most reliability-critical. The job is this: receive an HTTP POST request, write the POST data to a file, and return HTTP 200 OK. Anybody who has any server-side experience at all can write this in their preferred language in about 10 minutes.

We intentionally want to make this phase as absolutely simplistic as possible. This is the phase that accepts logs from the limited-size kmsg buffer on the client and puts them somewhere persistent. It's nice to have real-time alerts, but if I have to choose between somewhat delayed alerts or randomly losing log messages when things get ugly, I'll have to accept the delayed alerts. Don't lose log messages! You'll regret it.

The best way to not lose messages is to minimize the work done by your log receiver. So we did. It receives the uploaded log file chunk and appends it to a file, and that's it. The "file" is actually in a cloud storage system that's more-or-less like S3. When I explained this to someone, they asked why we didn't put it in a Bigtable-like thing or some other database, because isn't a filesystem kinda cheesy? No, it's not cheesy, it's simple. Simple things don't break. Our friends on the "let's use structured events to make metrics" team streamed those events straight into a database, and it broke all the time, because databases have configuration options and you inevitably set those options wrong, and it'll fall over under heavy load, and you won't find out until you're right in the middle of an emergency and you really want to see those logs. Or events.

Of course, the file storage service we used was encrypted-at-rest, heavily audited, and auto-deleted files after N days. When you're a megacorporation, you have whole teams of people dedicated to making sure you don't screw this up. They will find you. Best not to annoy them.

We had to add one extra feature, which was authentication. It's not okay for random people on the Internet to be able to impersonate your devices and spam your logs - at least without putting some work into it. For device authentication, we used the rarely-used HTTP client-side certificates option and the devcert program (linked above) so that the client and server could mutually authenticate each other. The server didn't check the certificates against a certification authority (CA), like web clients usually do; instead, it had a database with a whitelist of exactly which certs we're allowing today. So in case someone stole a device cert and started screwing around, we could remove their cert from the whitelist and not worry about CRL bugs and latencies and whatnot.

Unfortunately, because our log receiver was an internal app relying on internal infrastructure, it wasn't open sourced. But there really wasn't much there, honest. The first one was written in maybe 150 lines of python, and the replacement was rewritten in slightly more lines of Go. No problem.

Retries and floods

Of course, things don't always go smoothly. If you're an ISP, the least easy thing is dealing with cases where a whole neighbourhood gets disconnected, either because of a power loss or because someone cut the fiber Internet feed to the neighbourhood.

Now, disconnections are not such a big deal for logs processing - you don't have any. But reconnection is a really big deal. Now you have tens or hundreds of thousands of your devices coming back online at once, and a) they have accumulated a lot more log messages than they usually do, since they couldn't upload them, and b) they all want to talk to your server at the same time. Uh oh.

Luckily, our system was designed carefully (uh... eventually it was), so it could handle these situations pretty smoothly:

The log uploader uses a backoff timer so that if it's been trying to upload for a while, it uploads less often. (However, the backoff timer was limited to no more than the usual inter-upload interval. I don't know why more people don't do this. It's rather silly for your system to wait longer between uploads in a failure situation than it would in a success situation. This is especially true with logs, where when things come back online, you want a status update now. And clearly your servers have enough capacity to handle uploads at the usual rate, because they usually don't crash. Sorry if I sound defensive here, but I had to have this argument a few times with a few SREs. I understand why limiting the backoff period isn't always the right move. It's the right move here.)

Less obviously, even under normal conditions, the log uploader uses a randomized interval between uploads. This avoids traffic spikes where, after the Internet comes back online, everybody uploads again exactly 60 seconds later, and so on.

The log upload client understands the idea that the server can't accept its request right now. It has to, anyway, because if the Internet goes down, there's no server. So it treats server errors exactly like it treats lack of connectivity. And luckily, log uploading is not really an "interactive" priority task, so it's okay to sacrifice latency when things get bad. Users won't notice. And apparently our network is down, so the admins already noticed.

The /dev/kmsg buffer was configured for the longest reasonable outage we could expect, so that it wouldn't overflow during "typical" downtime. Of course, there's a judgement call here. But the truth is, if you're having system-wide downtime, what the individual devices were doing during that downtime is not usually what you care about. So you only need to handle, say, the 90th percentile of downtime. Safely ignore the black swans for once.

The log receiver aggressively rejects requests that come faster than its ability to write files to disk. Since the clients know how to retry with a delay, this allows us to smooth out bursty traffic without needing to either over-provision the servers or lose log messages.

(Pro tip, learned the hard way: if you're writing a log receiver in Go, don't do the obvious thing and fire off a goroutine for every incoming request. You'll run out of memory. Define a maximum number of threads you're willing to handle at once, and limit your request handling to that. It's okay to set this value low, just to be safe: remember, the uploader clients will come back later.)

Okay! Now our (unstructured) logs from all our 100,000 devices are sitting safely in a big distributed filesystem. We have a little load-balanced, multi-homed cluster of log receivers accepting the uploads, and they're so simple that they should pretty much never die, and even if they do because we did something dumb (treacherous, treacherous goroutines!), the clients will try again.

What might not be obvious is this: our reliability, persistence, and scaling problems are solved. Or rather, as long as we have enough log receiver instances to handle all our devices, and enough disk quota to store all our logs, we will never again lose a log message.

That means the rest of our pipeline can be best-effort, complicated, and frequently exploding. And that's a good thing, because we're going to start using more off-the-shelf stuff, we're going to let random developers reconfigure the filtering rules, and we're not going to bother to configure it with any redundancy.

Grinding the logs

The next step is to take our unstructured logs and try to understand them. In other words, we want to add some structure. Basically we want to look for lines that are "interesting" and parse out the "interesting" data and produce a stream of events, each with a set of labels describing what categories they apply to.

Note that, other than this phase, there is little difference between how you'd design a structured event reporting pipeline and a log pipeline. You still need to collect the events. You still (if you're like me) need to persist your events across kernel panics. You still need to retry uploading them if your network gets partitioned. You still need the receivers to handle overloading, burstiness, and retries. You still would like to stream them (if your infrastructure can handle it) rather than uploading every 60 seconds. You still want to be able to handle a high volume of them. You're just uploading a structured blob instead of an unstructured blob.

Okay. Fine. If you want to upload structured blobs, go for it. It's just an HTTP POST that appends to a file. Nobody's stopping you. Just please try to follow my advice when designing the parts of the pipeline before and after this phase, because otherwise I guarantee you'll be sad eventually.

Anyway, if you're staying with me, now we have to parse our unstructured logs. What's really cool - what makes this a killer design compared to starting with structured events in the first place - is that we can, at any time, change our minds about how to parse the logs, without redeploying all the software that produces them.

This turns out to be amazingly handy. It's so amazingly handy that nobody believes me. Even I didn't believe me until I experienced it; I was sure, in the beginning, that the unstructured logs were only temporary and we'd initially use them to figure out what structured events we wanted to record, and then modify the software to send those, then phase out the logs over time. This never happened. We never settled down. Every week, or at least every month, there was some new problem which the existing "structured" events weren't configured to catch, but which, upon investigating, we realized we could diagnose and measure from the existing log message stream. And so we did!

Now, I have to put this in perspective. Someone probably told you that log messages are too slow, or too big, or too hard to read, or too hard to use, or you should use them while debugging and then delete them. All those people were living in the past and they didn't have a fancy log pipeline. Computers are really, really fast now. Storage is really, really cheap.

So we let it all out. Our devices produced an average of 50 MB of (uncompressed) logs per day, each. For the baseline 100,000 devices that we discussed above, that's about 5TB of logs per day. Ignoring compression, how much does it cost to store, say, 60 days of logs in S3 at 5TB per day? "Who cares," that's how much. You're amortizing it over 100,000 devices. Heck, a lot of those devices were DVRs, each with 2TB of storage. With 100,000 DVRs, that's 200,000 TB of storage. Another 300 is literally a rounding error (like, smaller than if I can't remember if it's really 2TB or 2TiB or what).

Our systems barfed up logs vigorously and continuously, like a non-drunken non-sailor with seasickness. And it was beautiful.

(By the way, now would be a good time to mention some things we didn't log: personally identifiable information or information about people's Internet usage habits. These were diagnostic logs for running the network and detecting hardware/software failures. We didn't track what you did with the network. That was an intentional decision from day 1.)

(Also, this is why I think all those log processing services are so badly overpriced. I wanna store 50 MB per device, for lots of devices. I need to pay S3 rates for that, not a million dollars a gigabyte. If I have to overpay for storage, I'll have to start writing fewer logs. I love my logs. I need my logs. I know you're just storing it in S3 anyway. You probably get a volume discount! Let's be realistic.)

But the grinding, though

Oh right. So the big box labeled "Grinder" in my diagram was, in fact, just one single virtual machine, for a long time. It lasted like that for much longer than we expected.

Whoa, how is that possible, you ask?

Well, at 5TB per day per 100,000 devices, that's an average of 57 MBytes per second. And remember, burstiness has already been absorbed by our carefully written log receivers and clients, so we'll just grind these logs as fast as they arrive or as fast as we can, and if there are fluctuations, they'll average out. Admittedly, some parts of the day are busier than others. Let's say 80 MBytes per second at peak.

80 MBytes per second? My laptop can do that on its spinning disk. I don't even need an SSD! 80 MBytes per second is a toy.

And of course, it's not just one spinning disk. The data itself is stored on some fancy heavily-engineered distributed filesystem that I didn't have to design. Assuming there are no, er, collossal, failures in provisioning (no comment), there's no reason we shouldn't be able to read files at a rate that saturates the network interface available to our machine. Surely that's at least 10 Gbps (~1 GByte/sec) nowadays, which is 12.5 of those. 1.25 million devices, all processed by a single grinder.

Of course you'll probably need to use a few CPU cores. And the more work you do per log entry, the slower it'll get. But these estimates aren't too far off what we could handle.

And yeah, sometimes that VM gets randomly killed by the cluster's Star Trek-esque hive mind for no reason. It doesn't matter, because the input data was already persisted by the log receivers. Just start a new grinder and pick up where you left off. You'll have to be able to handle process restarts no matter what. And that's a lot easier than trying to make a distributed system you didn't need.

As for what the grinder actually does? Anything you want. But it's basically the "map" phase in a mapreduce. It reads the data in one side, does some stuff to it, and writes out postprocessed stuff on the other side. Use your imagination. And if you want to write more kinds of mappers, you can run them, either alongside the original Grinder or downstream from it.

Our Grinder mostly just ran regexes and put out structures (technically protobufs) that were basically sets of key-value pairs.

(For some reason, when I search the Internet for "streaming mapreduce," I don't get programs that do this real-time processing of lots of files as they get written. Instead, I seem to get batch-oriented mapreduce clones that happen to read from stdin, which is a stream. I guess. But... well, now you've wasted some perfectly good words that could have meant something. So okay, too bad, it's a Grinder. Sue me.)

Reducers and Indexers

Once you have a bunch of structured events... well, I'm not going to explain that in a lot of detail, because it's been written about a lot.

You probably want to aggregate them a bit - eg. to count up reboots across multiple devices, rather than storing each event for each device separately - and dump them into a time-series database. Perhaps you want to save and postprocess the results in a monitoring system named after Queen Elizabeth or her pet butterfly. Whatever. Plug in your favourite.

What you probably think you want to do, but it turns out you rarely need, is full-text indexing. People just don't grep the logs across 100,000 devices all that often. I mean, it's kinda nice to have. But it doesn't have to be instantaneous. You can plug in your favourite full text indexer if you like. But most of the time, just an occasional big parallel grep (perhaps using your favourite mapreduce clone or something more modern... or possibly just using grep) of a subset of the logs is sufficient.

(If you don't have too many devices, even a serial grep can be fine. Remember, a decent cloud computer should be able to read through ~1 GByte/sec, no problem. How much are you paying for someone to run some bloaty full-text indexer on all your logs, to save a few milliseconds per grep?)

I mean, run a full text indexer if you want. The files are right there. Don't let me stop you.

On the other hand, being able to retrieve the exact series of logs - let's call it the "narrative" - from a particular time period across a subset of devices turns out to be super useful. A mini-indexer that just remembers which logs from which devices ended up in which files at which offsets is nice to have. Someone else on our team built one of those eventually (once we grew so much that our parallel grep started taking minutes instead of seconds), and it was very nice.

And then you can build your dashboards

Once you've reduced, aggregated, and indexed your events into your favourite output files and databases, you can read those databases to build very fast-running dashboards. They're fast because the data has been preprocessed in mostly-real time.

As I mentioned above, we had our pipeline reading the input files as fast as they could come in, so the receive+grind+reduce+index phase only took a few tens of milliseconds. If your pipeline isn't that fast, ask somebody why. I bet their program is written in java and/or has a lot of sleep() statements or batch cron jobs with intervals measured in minutes.

Again here, I'm not going to recommend a dashboard tool. There are millions of articles and blog posts about that. Pick one, or many.

In conclusion

Please, please, steal these ideas. Make your log and event processing as stable as our small team made our log processing. Don't fight over structured vs unstructured; if you can't agree, just log them both.

Don't put up with weird lags and limits in your infrastructure. We made 50MB/day/device work for a lot of devices, and real-time mapreduced them all on a single VM. If we can do that, then you can make it work for a few hundreds, or a few thousands, of container instances. Don't let anyone tell you you can't. Do the math: of course you can.

Epilogue

Eventually our team's log processing system evolved to become the primary monitoring and alerting infrastructure for our ISP. Rather than alerting on behaviour of individual core routers, it turned out that the end-to-end behaviour observed by devices in the field were a better way to detect virtually any problem. Alert on symptoms, not causes, as the SREs like to say. Who has the symptoms? End users.

We had our devices ping different internal servers periodically and log the round trip times; in aggregate, we had an amazing view of overloading, packet loss, bufferbloat, and poor backbone routing decisions, across the entire fleet, across every port of every switch. We could tell which was better, IPv4 or IPv6. (It's always IPv4. Almost everyone spends more time optimizing their IPv4 routes and peering. Sorry, but it's true.)

We detected some weird configuration problems with the DNS servers in one city by comparing the 90th percentile latency of DNS lookups across all the devices in every city.

We diagnosed a manufacturing defect in a particular batch of devices, just based on their CPU temperature curves and fan speeds.

We worked with our CPU vendor to find and work around a bug in their cache coherency, because we spotted a kernel panic that would happen randomly every 10,000 CPU-hours, but for every 100,000 devices, that's still 10 times per hour of potential clues.

...and it sure was good for detecting power failures.

Anyway. Log more stuff. Collect those logs. Let it flow. Trust me.

Update 2019-04-26: So, uh, I might have lied in the title when I said you can't have this logs pipeline. Based on a lot of positive feedback from people who read this blog post, I ended up starting a company that might be able to help you with your logs problems. We're building pipelines that are very similar to what's described here. If you're interested in being an early user and helping us shape the product direction, email me!