Atenção! Esta oferta já expirou, não sendo possível efectuar mais candidaturas. Emprego Distributed Systems Engineer (M/F) Empresa:Landing.Jobs Data Publicação:13/08/2019 Descrição da Função:We are looking for talented profiles to help build and maintain the distributed data collection system that is at the heart of our business.We are a data-driven company which collects and processes more than 500GB of raw data daily. We leverage big data technologies such as Serverless, Spark on AWS EMR to crunch these volumes of data and make it queryable.In this role, you will ensure that our data collection engine, which consists of distributed web crawlers, is state of the art and ahead of our competition. You will ensure that we can scrape any webshop, no matter the ban-detection that has been put in place. Then, it will be important that proper monitoring tools are in place. We are currently scraping 60 sites and your goal is to at least triple that without losing completeness and quality.Your responsibilities will include:Creating and implementing Distributed web crawling architecturesImplementing cost-effective data processing architecturesCreating advanced system monitoring solutions & dashboardsDesigning advanced ways of interpreting scraped HTMLsManaging advanced proxies Requisitos: At least 5 years of experience in object-oriented software engineering & design in any object-oriented programming languageExperience with and understanding of large-scale web crawlingExperience with databases, SQLExperience with infrastructure such as load-balancers, cachesHighly proficient in spoken and written EnglishYou never stop learningNice to have:Have experience building on top of Amazon Web ServicesHave programming experience with PythonExpert knowledge of web-scraping & web-scraping architecturesExperience with GoLang & JavaScript (Node.js) is a plusExperience with big data technologies (such as Hadoop, Spark, Airflow, Cassandra, Elasticsearch) is a plusHave a deep understanding of cloud possibilities and limitations in the areas of distributed systems, load balancing and networking, massive data storage, and securityGet energy from working in a highly complex and challenging startup environment with a high tech productKnowledge of DevOps & automation (Terraform, Ansible)Data analysis using Pandas (Python) Oferece-se:Work with the latest tech stackWe're a quickly growing company => you're personal growth can be huge too.Health Benefits – Comprehensive coverage for medical needsMeal allowance – Monthly meal card along with a fully stocked kitchen with enough coffee and fruits, along with monthly team drinks and dinnersWork-Life Balance – We trust you to know your schedule and work when you feel most productiveLearning and Development – Attend meet-up, conferences, and events that interest you and benefit your personal and career growth.Local:Lisboa (Distrito de Lisboa) Partilhar esta informação Enviar a oferta: Distributed Systems Engineer (M/F) Comentários Atenção! Esta oferta já expirou, não sendo possível efectuar mais candidaturas. Candidate-se a esta oferta Será redireccionado para o site da empresa