<|jdZddlmZddlmZmZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZdd lmZdd lmZed ZedGddZddZddZ d ddZddZy)uE当日抓取器 - 针对指定板块抓取当日含关键字的帖子) annotations) dataclassfield)datetime)ZoneInfo)config) ForumFetcher)parse_thread_posts)parse_topic_list) get_parser) PriceRecordz Asia/ShanghaiT)slotsc~eZdZUdZded<ded<ded<ded<ded<d ed <ded <ded <ee Zded<y) MatchedPostu$匹配关键字并解析后的帖子inttopic_idboard_idstrtitleurl author_namerlast_update_timematched_keyword post_text)default_factoryzlist[PriceRecord] coin_recordsN)__name__ __module__ __qualname____doc____annotations__rlistr&/opt/daily-market/src/daily_crawler.pyrrs@.MM J HN&+D&AL#Ar%rc>tjD] }||vs|cSy)u8返回命中的第一个关键字，未命中返回 NoneN)r KEYWORDS)textkws r&_match_keywordsr+"s&oo :I r%c d|S)Nz/index.asp?boardid=r$)rs r&_build_board_urlr-*s  ++r%cg}d}t|}||kr|d|} td|d|d |j| \} } td t | ||j j } td t| d | s |S| Dcgc]5}|jjtjd |k(r|7}}td t|d|s |S|D]}t|j}|std|d|jdd |j|j\}} t!||j j |j"|j$|j&}|r|dj(nd}|st|}|xs|}|r|j+|||j|j|j|j$|d|j-t/|j||j|j|j$|j||[t1|}g}|r ||}t j4j|d}t7j8tj;}|D]}|j=id|d|d|d|jd|jd|jd |j$d!|j>d"|j@d#|jBd$|jDd%|jFd&|jHd'|jJd(|jLd)|jNd*|jPd+|i|j-t/|j||j|j|j$|j|||, |dz }||kr|S#t$r} td|d|d| Yd} ~ |Sd} ~ wwxYwcc}w#t$r&} td|jd| Yd} ~ Ud} ~ wwxYw#t2$r&} td|jd| Yd} ~ d} ~ wwxYw)-uy抓取指定板块当天的匹配帖子。 sample_mode=True 时仅存原始帖子正文，不做价格解析。 rz&page=u[crawl] 抓取板块u第u页...u [warn] 板块u页抓取失败: Nu[crawl] 解析主题列表...u[crawl] 发现u 个主题z%Y-%m-%du[crawl] 当日主题: u个u[crawl] 命中标题关键字 'z': (u#[warn] 主题抓取失败 topic_id=: )htmlbase_urldefault_author_keydefault_author_namedefault_profile_urlr) trade_daterrrrrr)rrrrrrrru[warn] 解析失败 topic_id=r7r board_typer topic_title topic_urlrname ref_price price_detailsunitquantity quantity_unit package_req quality_req other_notesraw_text fetched_at) rrrrrrrrr))r-printget RuntimeErrorr r r2lenr astimezone ASIA_SHANGHAIstrftimer+rrrr author_keyr profile_urlr) save_raw_postappendrr Exception BOARD_TYPErnow isoformatsave_price_recordr;r<r=r>r?r@rArBrCrD)fetcherdbrr7 sample_mode max_pages matched_postspage board_urlpage_urlr1_exctopicst today_topicstopictitle_kw topic_htmlpostsr content_kw matched_kw parser_funcrecordsr8rErecs r&crawl_board_todayrl.s(*M D *I ) [tf- $XJc$v>? kk(+GD! -/!$'..2I2IJ s6{m956 b [ !!,,];DDZPT^^    &s<'8&9=> P M"b E&u{{3H 3H:SSbAQ@RS T  ' EII 6 A ' 00#(#3#3$)$5$5$)$5$5 E*/a BI)3J#/xJ  ", ( %"[[ 99#(#4#4!*"$$["^^%++ % 1 1%*%;%;$.' & %X.K)+GS))4G  **..x>@J $$& *&&!*& & "5;; &   &"5#4#4&CHH& &$S%6%6&CHH& &$S%6%6&"3??&"3??& "3??!&" #&$!*%& ,  !kkII!--!&!7!7 *#$ " qb H  y ) | s  M(3tf4EcUK L n s  $  ;ENN;K2cUST d!S9%..9IC5QRRSsMO':P'PQ ' P0P  P Q QQ Q8Q33Q8c ddl}|j|d|jj|d|d}t |dd 5}t |d D]\}}|j d d |j d |d|jd |j d|jd |j d|jd |j d|jd |j d|jd |j dd |j |jdz ddd|S#1swY|SxYw)uB将匹配帖子正文写入样本文件，供设计解析器参考rNT)exist_okboardz _samples.txtwzutf-8)encodingrz<============================================================ u样本 r0zURL: u作者: u命中关键字: u更新时间: z<------------------------------------------------------------z ) osmakedirspathjoinopen enumeratewriterrrrrr)rZr sample_dirrsrufiposts r&save_sample_filer~s> KK TK* 77<< eH:\$B CD dC' * -a 2 -GAt GGxjO $ GGgaS4::,b1 2 GGeDHH:R( ) GGht//03 4 GG'(<(<'=R@ A GGnT%:%:$;2> ? GGxjO $ GGDNNV+ , - - K - Ks C9E  EN)r)rreturnz str | None)rrrr)F ) rVr rrr7rrXboolrYrrlist[MatchedPost])rZrrrrzrrr)r! __future__r dataclassesrrrzoneinforr6r rVr forum_threadr forum_topicsr parsersr parsers.baserrKrr+r-rlr~r$r%r&rsK"(!,*%)   B B B, N NN N  N  NNbr%