Recently I got my pendrive infected with the infamous shortcut virus from a local print shop. I decided to investigate on the workings of this virus, and surprisingly, I could not find any resources on the internet that actually talk about “How” this virus works. Yes, there are dozens of tutorials telling you how to get rid of the virus, but none that tells what exactly is it doing, or how to analyze it. That’s the motivation for this post. Hopefully, by the end, you’ll have some knowledge of how Windows works, and how this virus tricks you into becoming infected.
Just wanted to write this down because I thought it might be helpful to some other folks.
Recently I’ve been looking into ranged queries and solving related CP problems. There is this trick called Square root decomposition which turns a query time from to with some pre-processing
There are lots of good tutorials out there for square root decomposition, so I won’t touch that here, just would like to note that during implementation, in a ranged query from a to b, we do not have to sweep through all the blocks. We can further speed up the query by determining the leftmost block and rightmost block (this can be done by dividing a and b by the size of a block) and then querying between them.
Now why would this be so useful? There are questions on some OJs where time limits are tight. For e.g this leetcode problem https://leetcode.com/problems/range-frequency-queries/ has several ways of solving it. I used square root decomposition for this, however the time limits and test cases are such that without the above optimisation, the solution may not get Accepted always.
What we’re doing above is just reducing the constant factor in big-O notation, thereby reducing the actual time taken 😉
I wasted quite some time on this silly thing, before realizing why my square root decomposition solution was not passing. Now I know what not to do when implementing square root decomposition 🙂 .
If you’re reading this, I’d highly encourage you to try out the problem once and implement square root decomposition to observe it for yourself :D.
Often when programming front-end for applications in popular JS frameworks like React-JS, I come across the concise, but hard to process (initially) ... operator. Once I understood the functionalities, it’s actually extremely useful for writing things compactly, and sometimes the only way, but what does it exactly mean always confused me in the beginning, because it can take up several meanings depending on the context.
The common name for this operator is either “spread” operator or “rest” operator, depending on the context. For clarity on this, let’s take simple JS code snippets.
I’ve been using a dual booted Ubuntu 18.04 with Windows 10 for roughly a year and half now and as usual, follow the guides on the internet for dual booting. However, there are issues that no guide talks about and I think they’re pretty important too. This post is dedicated to such issues and their solution as faced by me.
Disclaimer: You need to have an active Vodafone mobile prepaid or postpaid subscription in order for this to work. From my knowledge, Airtel also offers free Zee5 subscription, however I don’t have an airtel subscription, so I cannot confirm.
Also note that as per the website, this offer is valid only till 31st January 2021, so these instructions might not work post that date anymore.
Onto the main attraction now!
Recently my family bought a firestick, to convert our old 2013 Sony TV into a smart TV with least replacements. So far, we haven’t experienced much issues, except a few favorite android apps are not available on the Firestick-TV Store.
Since my family has a vodafone mobile number, we got this complimentary Zee5 subscription of Vodafone play, which provides us access to Zee5 free of cost.
However, in order to open Zee5 one needs to have the Vodafone play app installed. While Vodafone Play does have some extra content, most of the content is derived from Zee5 network. Unfortunately, in order to access the Zee5 content via the complimentary offer, one needs to open it through the Vodafone Play App, which is not available on the Amazon App store.
That’s a bummer, but not to worry! Zee5 has an app for TV, and we can run our complimentary subscription by taking advantage of our phone! The steps are relatively simple:
Install Zee5 app on your firestick, keep it open and select login and leave it there
Install Zee5 app on your android phone (I am using version 16.27.14), don’t open it yet
Install Vodafone Play app and open it
Login using your Vodafone number and OTP
Open any Zee5 media content, this should open the Zee5 app and sign you in automatically
Now, minimize the video screen by dragging it to the bottom. This is the important part.
Now, you should be able to access all content on Zee5. No big deal though, you could do this through Vodafone play app as well. However, now you can also access the settings portion!
Open the menu bar (hamburger icon on the top usually), and select Authenticate
Enter the code that is shown on your TV’s app and continue
Now continue on the TV screen as well
And done! Although the above might look like a lot of steps, they really very simple when you try them. The important part is ability to explore whole of Zee5 App. If you are not able to minimize the video screen, I’d recomment installing an older version of the app (I’ve tested it on 16.27.14, and it works). Also, these steps should work on any android TV, however I’ve only tested it on the generation 2 Firestick.
We can now use our complimentary Zee5 subscription offer on an Android TV as well!
Node embeddings have brought a whole new dimension to compressing graphs and efficient inferences over graphs. Another interesting perspective is node embeddings of graphs that evolve over time!
During my last semester, I chose a project to investigate how much of a difference can static embeddings make over dynamic embeddings. For this task, I developed my own static embedding models and compared them with a fairly recent dynamic embedding model called JODIE.
First things first, while going through the code of JODIE, I realized that the equations in the paper are not in sync with the equations implemented in code. More specifically, the paper defines update operation as:
However, in code, this is not the case. Inside code, it is closer to the following:
From first equation, I inferred that if the code is able to succesfully handle that many parameters with an RNN Cell, then probably I can go one step further and instead of learning , I could predict these matrices from an abstract 3-D tensor (this methodology was developed after discussion with another PhD Student, Subhabrata). This post, will not discuss the modified methodology though. We’ll stick to original JODIE implementation.
Just mentioned this bit of information, because the new method fell flat during implementation. Anyways, with failures out of the way, I’ll dedicate the rest of this post in discussing how JODIE works.
First off, before reviewing what the paper proposes, let’s understand basic jargon. We use something called as interaction networks. These are also a kind of graph data, however they have a timestamped information about when did a particular user interact with a particular item. This information is neatly represented in a tabular fashion. Hence the input files are CSV files with one interaction per row. Every row has an information about user, item, timestamp and the features of that interaction.
I won’t go into how the loading of dataset works, however, I’d like to mention the T-Batching algorithm that is in use here. This is a relatively simple way to parallelize the processing of interactions. We cannot process two interactions that involve either the same user or the same item simulatneously. Hence, the authors proposed this algorithm in order to create T-Batches out of interactions so that all interactions within a T-Batch can be processed at once.
The algorithm roughly works as follows:
lib = all tbatches set to empty using a dictionary
for j in interactions:
userid = user_sequence_id[j]
itemid = item_sequence_id[j]
feature = feature_sequence[j]
user_timediff = user_timediffs_sequence[j]
item_timediff = item_timediffs_sequence[j]
tbatch_to_insert = max(lib.tbatchid_user[userid], lib.tbatchid_item[itemid]) + 1
lib.tbatchid_user[userid] = tbatch_to_insert
lib.tbatchid_item[itemid] = tbatch_to_insert
lib.current_tbatches_user[tbatch_to_insert].append(userid)
lib.current_tbatches_item[tbatch_to_insert].append(itemid)
lib.current_tbatches_feature[tbatch_to_insert].append(feature)
lib.current_tbatches_interactionids[tbatch_to_insert].append(j)
lib.current_tbatches_user_timediffs[tbatch_to_insert].append(user_timediff)
lib.current_tbatches_item_timediffs[tbatch_to_insert].append(item_timediff)
lib.current_tbatches_previous_item[tbatch_to_insert].append(user_previous_itemid_sequence[j])
timestamp = timestamp_sequence[j]
if tbatch_start_time is None:
tbatch_start_time = timestamp
if timestamp - tbatch_start_time > tbatch_timespan:
tbatch_start_time = timestamp # RESET START TIME FOR THE NEXT TBATCHES
##### Train model by iterating over everything in the tbatch dictionary
##### reset the tbatch dictionary
In the T-Batching algorithm, note that the main line is “`tbatch_to_insert = max(lib.tbatchid_user[userid], lib.tbatchid_item[itemid]) + 1“`. All that’s done is a mapping is maintained for all users and items about the last tbatch that they were inserted in. Then 1 is added to the max amongst both of them to allocate a different “bucket”. That’s really neat and nice!
Now on the main algorithm, it may look a bit convoluted at first look, but really is a simple improvement over the previously existing approaches. Essentially, the following points happen:
Learning of User and Item Embedding
Prediction of User embedding
Prediction of Item embedding
For 1, a single Elman RNN cell each for user and item is trained, by feeding the user and item features as described by the modified equations above. The output of these networks will be the actual user and item embedding at current time step.
For 2, the time difference is sent to another trainable single layer neural network which converts time into a time context vector. This time context vector is multiplied with user embedding at previous time step, in order apply some kind of displacement over the user’s embedding and predict their embedding at current time step.
For 3, the predicted user embedding and previously interacted item’s embedding are concatenated, along with their static embeddings and then fed into another neural network. This predicts the embedding of the item that the user is likely to interact with next.
How are all these layers trained? They are simply trained with two losses. First loss is exactly corresponding to our task, i.e predicting the item embedding. We have our actual item embedding from step 1 and predicted item embedding from step 3. We take an MSELoss of both of these.
The second loss is used to maintain continuity over any two consecutive embeddings. This is done by minimizing the loss between user embedding at previous step and user embedding at current step, and similarly item embedding from previous step and item embedding at current step.
And that’s pretty much it!
I hope this demystification of JODIE algorithm will help someone working on this in future. Note that I have only explained the case of next item prediction, it doesn’t cover the case of state change prediction. However, the steps do not change much, all that is done in order to add supervision for state change prediction is just another loss which computes cross entropy between currently predicted state and actual state.
Thanks for reading!
References: The equation figures have been rendered from latex.
Recently I recieved a request from someone on how to install the NSE EOD Data Downloader software that I had updated in January 2020. This made me realise that the users of this software are probably end users who do not have programming knowledge / experience and I should probably keep their interest in mind as well with the update.
Thus, I am providing a small tutorial to help them set-up the emergency release software and use it. Please let me know in the comments if you find any faults or some of the steps don’t work for you. Also bear in mind that I do not own a Windows PC, so all the screenshots are taken on a windows virtual machine. I hope they will resemble the actual machines closely, but if they don’t please let me know in the comments.
I’ll try to make an installer to make this process easier with future releases. However, I cannot promise that I’ll get enough time to work on this actively, thus I’m writing this tutorial for emergency releases.
I’ll list down the steps and then show them one by one:
Download the JRE (Java Runtime environment, so that you can run Java programs)
Download NSE-EOD data downloader zip file from https://github.com/virresh/nseeod/release. Extract the files using 7zip / winzip / winrar (I think new windows has option of doing this directly as wel).
Now onto the useful steps:
1) Downloading JRE
JRE is available free of cost on official oracle website. Download the correct file for your computer from here: https://www.java.com/en/download/ In case you are not sure about your platform, it is most likely Windows x64. I’m attaching screenshots for installing with offline downloader on Windows 7:
Once you’ve downloaded the jdk and reached the last dialogue box, you’ll get on-screen instructions. You have to click next and complete the installation.
2) Download NSE EOD Data Downloader
Goto https://github.com/virresh/nseeod/releases/latest. => Click on assets and expand it. => Download the NSEEODFixed.zip (This zip is provided only for release v3.3.1, I am yet to figure out a release pattern for future releases). => Extract with 7zip / winzip / winrar on desktop Here are some screenshots to explain the process:
And that’s it! Your data will be downloaded inside the respective folders. (e.g Equity data will be downloaded inside the Equity folder)
Your can choose your preferences in the settings. I intend it to work for atleast equities. Haven’t tested the other options. If you would like to help out with the development of this software, please head over to https://github.com/virresh/nseeod/issues and file a helpful issue / contribute to an existing issue.
Let me know in the comments in case you have issues.
While doing competitive programming problems, one of the most frequently encountered concept is of brute-forcing through solutions. People call it by many names – Complete Search, Brute force, backtracking, recursive search and whatnot. However, the core idea is same — Search through all possibilities till you arrive at an answer.
In this post, I’ll try to explain the art of writing brute-force and illustrate some ways in which one can write bruteforce. However, first I’d like to provide a generic template of how to think about any problem with the brute force approach.
The OSS world is a great place. We can find amazing softwares that people have written. However, the creators of most of these projects had made them voluntarily, and as time goes on, life happens and it becomes extremely difficult to maintain them. This is why a community is required for long-term sustenance of OSS projects. Big examples are like Apache and Mozilla.
My grandfather uses a similar oss software called NSE EOD Data Downloader. The original home of the software is http://nse-eod-downloader.blogspot.com/. The author did a nice job and published the software of various platforms like source-forge etc and even made the source code available on GitHub. However, since 2016, the code was no longer being maintained. In 2020 came the blow that rendered this software useless.
I’ve had my fair share of experience with basic security tasks in traditional languages. The first and foremost topic that I got introduced to was buffer overflow. However, many tutorials on this simple technique fall short of one thing or the other and I end up searching for the missing parts in some other tutorial. I’d like to collate all the resources here in one place, for anybody who comes looking for these. Another reason for writing this post is that most existing tutorials focus on 32-bit systems, however the age of 32-bit systems is long gone. These days one can hardly find a 32-bit only system accessible to them (although I’m not sure about production systems). Due to the prevalance of 64-bit systems, I found several tutorials hard to read. I hope this post will help someone looking to get started with buffer overflows (extremely common under the PWN section of many CTFs). So brace up, this will be a long one!
End goal of this post: Make a vulnerable binary print “Hello World!”.
First off, how does this attack work? 1) You somehow gain the ability to overwrite the return address of a function 2) You exploit the vulnerability to overwrite this return address to a piece of code that does what you want. It’s possible that this code already exists in the memory and you just have to execute it.
So, what do you need for this attack? The answer depends on the kind of vulnerability you have. For the purpose of demonstration, I’ll illustrate buffer overflow on a program that allows execution of code from stack.
to 
with some 
blocks. We can further speed up the query by determining the leftmost block and rightmost block (this can be done by dividing a and b by the size of a block) and then querying between them.
, I could predict these matrices from an abstract 3-D tensor (this methodology was developed after discussion with another PhD Student, Subhabrata). This post, will not discuss the modified methodology though. We’ll stick to original JODIE implementation.
