• Breaking News

    Sunday, August 19, 2018

    Home Unlabelled Save short text fragments in text files or database Ask Programming

    Save short text fragments in text files or database Ask Programming

    August 19, 2018

    Save short text fragments in text files or database Ask Programming

    Save short text fragments in text files or database

    Posted: 19 Aug 2018 09:08 PM PDT

    Let's say I have a huge list of short text fragments (around 500 characters long each) similar to tweets to save to disk. Let's assume I have 1 billion fragments, and I have 500 Gb worth of pure text data.

    Method 1: Multiple Text Files, One Record Per File

    I am thinking of saving each of them to a text file for simplicity, naming them using a incrementing ID number like "1.txt ... n.txt". I can easily inspect them using type or cat command, search through them using standard command line tools like grep. Except if I want to search by particular field e.g. author, I may have to open up each file, parse it and scan the line/position containing the data I want for matching. For a few thousand records, it may work fine, but once it goes to a 100 thousands or million records, it may get slow.

    The bad thing I foresee is the number of files created, and I wonder if the file system works well (retrieving and searching) with many small text files. If I am not mistaken, each file would take up a cluster with 4 Kb size at the minimum, meaning a lot of wasted unused space. And probably the disk have to skip around a lot.

    Method 2: One Text File, Multiple Records

    Another way is to save everything into 1 big text file, but each time I read it, it would be from line 1 to the last line. The startup processing will get slower as more data gets appended to the long text file. I probably would parse and put everything in memory if I do not have any sophisticated data management. So again, as the data grows, the memory usage and loading time grows.

    And there's no way I should or can read a 500 Gb text file all at once, Then I would probably have to deal with intelligently splitting them into parts into 1 Gb each and also knowing how to skip and load/read the right text file for a particular record. So this is another complication.

    Method 3: Database (Sql/NoSql)

    Another method is to stuff them into a (SQL/NoSQL) database with a varchar column. Metadata like author can probably be put into another column, so there is no need for parsing through each text fragment when there's a search or query.

    Does this sound like a MySQL database would be a better option than the text files?

    The bad thing is I would have to run a database server process and keep it running and listening for a connection. Furthermore, if it's encoded in some pretty proprietary format, I would have to write custom scripts and queries to inspect the saved data.

    I am also thinking if MySQL is not a good fit, would the good old dbm that is widely available in many programming languages a good alternative.

    I know this sounds like the classic file or database storage question that has been discussed to death, but I would like to hear any helpful comments.

    submitted by /u/2048b
    [link] [comments]

    Im starting as a GTA for a systems programming lab tomorrow. Last minute advice?

    Posted: 19 Aug 2018 05:27 PM PDT

    Im well versed on the subject and have some teaching experience, but I'd like to hear ideas and advice from other TAs, professors, and students on how to create a chill and helpful learning environment.

    submitted by /u/citra-ceth
    [link] [comments]

    I need advice on where to start

    Posted: 19 Aug 2018 05:22 PM PDT

    I'd like to make my life as a teacher easier if I can. Ive seen a neat classroom management system but I dont think I can implement it without some coding. I have a little experience with coding but haven't made anything lile what I'm hoping to make for this.

    I would need a program to look at an email sent to my account, match the senders email to a student in a seperate database, read the subject line to decide the action (input a number from the subject line into a document in a folder named alo read from the subject line), and write a 0 for all students it doesnt recieve a message from.

    Ive only taken intro-level programming classes for C#, javascript, and HTML and dont have experience making something that interacts with other programs. Is this a feasible undertaking for me?

    submitted by /u/werdswerdswerds
    [link] [comments]

    Legit form grabbing?

    Posted: 19 Aug 2018 04:45 PM PDT

    I need to capture data as it is entered into web forms by employees and push it into a database. Employees would definitely know this is happening and it's only for a few selected sites. They are placing orders from a few sites and we want to capture that data directly as it is entered. I'm aware of form-grabbing malware but I'm looking for a legitimate, legal, commercial application that we can run in the browser. Does anyone know of such a beast?

    submitted by /u/TinyE33
    [link] [comments]

    Help dealing with bad freelancer on API project?

    Posted: 19 Aug 2018 08:06 AM PDT

    Hello, I had to contract a project for my non-profit out to a freelance programmer. Project was to build an host an API plus interface for us to update a database that our partner organization queries. I'm not a CS person so don't quite understand the words I'm saying. Regardless, he's been terribly slow and after a month he still hasn't completed the job, despite quoting three days completion. I've already paid for the chunk of work he's done, and I don't want to lose that. I'd like to take the chunk he's done and have a friend finish it up for me. I'd also like to write him a terrible review on the freelancing site. In the latter case, I'm worried that if he sees a bad review, he may stop hosting, rescind my access and not give me access to the code. All I have thus far is login access to the hosted database interface he created. My questions: 1) what should I do? 2) can I ask him to send me the code? - what should I ask him to send me in order for a friend to build onto what is already done? Thank you!

    submitted by /u/plesam412
    [link] [comments]

    I have a huge .sql file that's too large to be put in my usual sql-to-json converters. What's the easiest way to do it?

    Posted: 19 Aug 2018 01:30 PM PDT

    [Mobile] Does camera integration in app development become more complex for dual camera and triple camera phones?

    Posted: 19 Aug 2018 05:15 PM PDT

    Please help me avoid Javascript.

    Posted: 19 Aug 2018 04:18 PM PDT

    I am looking to make a text based game that can run from a browser for maximum cross compatibility. It will involve processing lots of objects. At the moment I'm looking at Twine with Sugarcube but that involves Javascript and I hate Javascript.

    So dumb question: Is there something that allows me to use a language like python, java, c++ that somehow locally links to a browser? No servers or anything. Just a file that you open with your browser and displays a nice html/CSS page but clicking on something activates some local object oriented logic that updates the page.

    submitted by /u/new_mode
    [link] [comments]

    You have to make an internal tool for your job and it needs a GUI. What technology would you choose to develop it?

    Posted: 19 Aug 2018 02:43 PM PDT

    C/C++ Programming Environment in Windows

    Posted: 19 Aug 2018 08:14 AM PDT

    Which IDE (Visual Studio, Cygwin, Eclipse) is more convenient for you for C/C++ programming on Windows?

    submitted by /u/before_i_die_alone
    [link] [comments]

    Looking for advice: switching from MognoDB on EC2 Instance to DynamoDB

    Posted: 19 Aug 2018 09:47 AM PDT

    Hello all,

    Context: I am running into some trouble using MongoDB, and looking to possibly move towards DynamoDB. Right now, I am currently hosting a MongoDB database running Mongo 4.0 using an EC2 Instance. It is a copy of our production database, which is on an identical EC2 instance, but is running Mongo 3.0. I currently use the copy to take in a dump from the production, and run an aggregation query on it. This aggregation query creates a new collection, using complex aggregations from three different collections. Hosting two EC2 instances costs a lot of money.

    I mainly get rid of extraneous data, change a data string to a date object (so I can query by date later on) and change a few strings to integers to allow for sorting.

    For simplicities sake, we'll call the production Database A and the copy Database B. Again, the only difference between the two is that MongoDB is upgraded. I want to upgrade production eventually to 4.0, but we are worried that it will mess with our application. In my testing, it doesn't seem to have an effect, but again, we don't want to ruin our application.

    I then use an AWS Lambda function, where I query the copied database, and perform more aggregation. Mainly, I query by a date range, and then group all of the data together. Data is grouped using a unique YoutubeId, and an Event Type (i.e, wedding, bar mitzvah etc). It counts how many times that youtubeId appears, the number of 'likes' the youtube link got from our users, and a few other stats. It then outputs this information to a CSV file, and uploads that to an S3 bucket. That allows us to download CSV files more than 6MB without worrying about Lambda limitations.

    The Question: My biggest question would be: is this possible in DynamoDB? I need to perform these advanced aggregations and basically output a new collection. I need to do sums, and transform data. It is a lot, but MongoDB has been good because its aggregation pipeline is so advanced.

    If that is not possible, how can I lower costs on Database B? It has a lot of extraneous collections on there, that really aren't necessary. It has 27 collections, but in reality, I only need to use 3. I can then eliminate those once I have the outputted aggregation of them, creating just one collection.

    submitted by /u/50mac50
    [link] [comments]

    Image Upload Issue on my Website

    Posted: 19 Aug 2018 09:21 AM PDT

    My website is simplecanvas.us

    It's actually my developer who made the site and is running into the issue. The issue: when we upload an image, I have to scroll down before the image shows up and actually loads. Any thoughts on how this can be fixed? Check the site out and try uploading to see the issue

    Thank you so much for looking at this. We've both been so stumped on this so I'm trying to help out.

    I can send you a screen video of what I'm seeing and the direct issue if you dm me your email.

    submitted by /u/derekrohn
    [link] [comments]

    Alternatives for Azure App Service free plan?

    Posted: 19 Aug 2018 12:57 PM PDT

    Hey, I'm wondering if there are other services that provide something similar to the Azure App Service free plan. This is an equivalent of AWS Elastic Beanstalk and at the free plan, Azure gives you:

    • 60 mins of CPU time
    • 1 GB RAM
    • 1 GB disk

    Is there something similar to host a Node.js app?

    I checked AWS but seems they don't have a free version of Elastic Beanstalk after the initial 12 months.

    submitted by /u/remember_that_girl
    [link] [comments]

    How to point to individual objects inside of an svg image on a webpage.

    Posted: 19 Aug 2018 10:37 AM PDT

    Forgive me because I am new if that question is wonky. What i am trying to do is be able to click a part of a picture, for instance a car and click the car door, and create a micro drop down. I have created an svg of a car and have the car door as a separate object but dont know how to point to it and have it create a drop down for them to place a number in it.

    submitted by /u/renderloading
    [link] [comments]

    Dynamic ebook archive

    Posted: 19 Aug 2018 08:47 AM PDT

    Is there a way for me to recursively take the names of a directory's contents & list them dynamically somewhere? I want to know how many ebooks I have & other metadata.

    submitted by /u/thundershr1ke
    [link] [comments]

    Any open-source screen magnifiers?

    Posted: 18 Aug 2018 11:01 PM PDT

    I am looking at making a screen magnifier, that can be toggled with a button and follows the mouse. If any source is out there, thanks.

    submitted by /u/HoelessJoe
    [link] [comments]

    What is scripted code walkthrough called? Why isn't is popular?

    Posted: 18 Aug 2018 10:55 PM PDT
    Read more
    at

    No comments:

    Post a Comment