202111

this cluster currently has [1000]/[1000] maximum shards open on logstash

[2021-11-26T15:47:31,250][WARN ][logstash.outputs.elasticsearch][elastiflow][0dba80b2c13f51c2e9dbdff0c01a879811b7fafc6ee06abbd3938ee05525180f] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"elastiflow-4.0.1-2021.11.26", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x278c2e35>], :response=>{"index"=>{"_index"=>"elastiflow-4.0.1-2021.11.26", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [6] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}}}}

{
  "index": {
    "lifecycle": {
      "name": "elastiflow"
    },
    "codec": "best_compression",
    "mapping": {
      "total_fields": {
        "limit": "5000"
      }
    },
    "refresh_interval": "10s",
    "number_of_shards": "1",
    "number_of_replicas": "1"
  }
}

rollover aliasの設定

  • 怒られる
Index lifecycle error
illegal_argument_exception: setting [index.lifecycle.rollover_alias] for index [elastiflow-4.0.1-2021.07.23] is empty or not defined

java.lang.IllegalArgumentException: setting [index.lifecycle.rollover_alias] for index [elastiflow-4.0.1-2021.07.23] is empty or not defined
  at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:75)
  at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:175)
  at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:335)
  at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:273)
  at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:184)
  at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:217)n
  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
  at java.base/java.lang.Thread.run(Thread.java:832)
  • 原因はrollover_aliasを設定していないのが原因．rolloverしたときにindexのaliasをどうするか，らしいので，indexのなまえをfollowする感じにしておくと同じ感覚で検索できるのでよさそうとのこと．
{
  "index": {
    "lifecycle": {
      "name": "elastiflow",
      "rollover_alias": "elastiflow"
    },
    "codec": "best_compression",
    "mapping": {
      "total_fields": {
        "limit": "5000"
      }
    },
    "refresh_interval": "10s",
    "number_of_shards": "1",
    "number_of_replicas": "1"
  }
}
  • bootstrapするindexのaliasを is_write_index を true にしてつける
curl -X PUT "localhost:9200/elastiflow-000001?pretty" -H 'Content-Type: application/json' -d'
{
  "aliases": {
    "elastiflow": {
      "is_write_index": true
    }
  }
}'

mutation testing

  • mutation testingについて考えた - Qiita
  • すでにテストが存在するコードに対してtestがfailするようなロジック変更を行い，実際に意図したとおりにtestがfailするかを確かめるもの．
  • 自動でやるtest frameworkも存在する．
  • logicをかえてtestを回しまくらないといけないので，それなりに時間的につらいものはありそう．

alpineなdockerでcronする

FROM alpine:3.9

RUN apk --update add tzdata && \
cp /usr/share/zoneinfo/Asia/Tokyo /etc/localtime && \
apk del tzdata && \
rm -rf /var/cache/apk/*

COPY crontab /var/spool/cron/crontabs/root
# cronで呼びたいbinとかをを適宜準備する
# COPY test /usr/local/bin/test

# cronのdaemonを起動(log level=1, foreground)
CMD crond -l 1 -f

正規表現の [\s\S] は . と違って改行文字にもマッチする

Einstein - Annalen der Physik

  • アインシュタインの原論文